DEV Community: Ahmet Küçükoğlu

What is Modular Monolith?

Ahmet Küçükoğlu — Wed, 09 Mar 2022 15:48:51 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/what-is-modular-monolith

Traditional Monolith

Firstly, let's talk about Traditional Monolith approach. This approach focuses on layers. It includes three layers, UI, Business and Data. All features in a project are vertically separated into these layers. Among those three layers, the business layer is the one that contains business logics of all features. Each feature knows business logic of other features, which is a fact we call tightly coupled.

Modular Monolith

The main focus of Modular Monolith is to separate modules. Each module has its own layers ( Domain, Infrastructure and API etc.). Thus, they can use different database solutions. On the other hand, modules don’t share their own business logics with each other. They can communicate with each other with sync or async approaches. These approaches are called loosely coupled.

Modular Monolith has a single binary. Therefore, apps that implement Modular Monolith can’t be scaled horizontally. Also modules are separated logically, not physically. So Modular Monolith doesn’t have a physically distributed structure. If we want to physically separate it, we have to implement the microservice approach.

Traditional Monolith isn’t a bad choice but it is likely to turn into a big ball of mud due to its structure. Its transition to microservice is also very difficult. On the other hand, Modular Monolith can easily migrate to microservice. Microservice is a good choice if your application will be published at internet scale. But Modular Monolith is a better option if your application will be published just within an organization.

Communication in Modular Monolith

Let’s consider an ordering system in an e-commerce app. Basic modules in an ordering system are Inventory, Order and Payment. During a buying process, order is received, stock is checked and payment is made. If a product is out of stock or payment hasn’t been made, that order is cancelled. Then how does such a communication occur in Modular Monolith?

There are two approaches (sync, async) for communication. In the sync approach, each module exposes an interface. These interfaces are defined in the shared layers of modules. The inventory module exposes IInventoryService which has the methods of ReserveStock, ReleaseStock. Likewise, the payment module exposes IPaymentService which has the MakePayment method. The order module communicates with other modules by using these interfaces.

In the async approach, each module raises events. These events are defined in the shared layers of modules. The order module raises the OrderReceived event. The inventory module raises the events of ReserveStock, OutOfStock and ReleaseStock. Likewise, the payment module raises the events of MakePayment, PaymentMade and PaymentFailed. Each module subscribes to related events. So the communication occurs through the message broker.

There are two approaches (choreography, orchestration) in async communication. It doesn’t matter which one is preferred. I preferred the orchestration approach because it is more efficient than the choreography approach for this scenario.

How is the project structure?

Bootstrapper is the entry point of the project. So it is a WebApp.

Each module has its own layers which aren’t standard. You can apply Onion or Hexagonal Architecture.

Shared.Abstractions : It is used by Modules.Core and Modules.Infrastructure. It contains interfaces, abstractions such as DDD, CQRS etc.
Shared : It is used for cross-cutting concern. It also includes implementations of the abstractions.

Ecommerce.Inventory : It is a class library. It contains the controllers of the inventory module. If you intend to use async method for communication, you can add the consumers (ReserveStockConsumer, ReleaseStockConsumer etc.) in Ecommerce.Inventory.
Ecommerce.Inventory.Core : It contains the domain or business logic of the inventory module. If you wish, you can separate Ecommerce.Inventory.Core into two parts, Application and Domain.
Ecommerce.Inventory.Infrastructure : It is where the modules are connected to external services or components such as database, message queue etc.
Ecommerce.Inventory.Shared : It is used to expose some classes (OutOfStock, StockReserved etc.) and allow the other layers to use those classes.

Summary

Using Modular Monolith is a better choice if you don’t want your project become a big ball of mud, and your project doesn’t require internet scale, or if you are likely to convert your project to a microservice architecture later.

You can access the sample projects from my Github address.

ahmetkucukoglu / modular-monolith-async

Ecommerce Modular Monolith App with Async

modular-monolith-async

Ecommerce Modular Monolith App with Async

Run the following command to start the project.

docker-compose up -d

Postman Collection

View on GitHub

ahmetkucukoglu / modular-monolith-sync

Ecommerce Modular Monolith App with Sync

modular-monolith-sync

Ecommerce Modular Monolith App with Sync

Run the following command to start the project.

docker-compose up -d

Postman Collection

View on GitHub

Good luck.

Developing Blockchain App – #1 Create Smart Contract

Ahmet Küçükoğlu — Wed, 14 Jul 2021 20:25:28 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/developing-blockchain-app-1-create-smart-contract/

This series of articles will be 3 parts. We will develop a blockchain app in this series of article. It will be like Hand-on Labs. So we prefer way of learning by developing an application. The subject of this article is to create a smart contract.

Smart Contract ve Solidity

Smart Contract is code parts that also store data in it. Programming language that used to write smart contract is Solidity. It isn't very advanced but it is enough to write.

Let's take an example to better understand smart contract. Shipping date is specified by sellers in the product description. We also order according to this date. But the product may not be shipped in this date. Or it may not be delivered by courier companies. If we look from the viewpoint of buyer, these two problems are not the seller's or the cargo company's. Because our interlocutor is the marketplace. If our order is shipped late or delivered late, we should be able to cancel it. But It is not possible to trust marketplace. Smart contract provide a nice solution. We create this business process on the smart contract and show the contract transparently. So we make sure that the process will be executed. We remember that a published contract can't be change.

Let's start to develop this contract. Our process will include below steps.

Own of the contract, so marketplace , will write an order on the contract.
Buyer will make payment for the order.
If the order isn't sent at specified date, buyer can cancel the order and get her payment back.

Preparing the Environment

Let's install Truffle which provides a development environment. We compile, test and publish of contracts by using Truffle.

npm install -g truffle

Let's install Ganache to create a local blockchain network on our machine. Ganache creates a local local blockchain network accessible on port 7545. This network have 10 accounts with balance of 100ETH. We will make test by using these accounts. You can install from the link below.

https://www.trufflesuite.com/ganache

Creating Smart Contract

Let's run the following command.

mkdir order-dapp-sample
cd order-dapp-sample
mkdir backend
cd backend
truffle init
code .

Application directory have contracts, migrations ve test directories. We create a contract named Orders.sol under the contracts directory.

We add a migration named 2_deploy_contracts.js under the migrations directory.

We add a test named order.js under the test directory.

We adjust the network section in the file truffle-config.js.

Developing Order Adding and Querying Functions

Marketplace will write OrderId, TotalPrice, DeliveryDate to the contract. So we create a struct named Order. It resembles struct in C#.

We store id as string type because Solidity don't have Guid type. Solidity have numeric types from uint8 to uint256. We use uint256 for totalPrice. Solidity don't have DateTime type. We have to store it in unix timestamp format. So We use uint256 for deliveryDate.

We add the createdDate for creation date of the order, the deliveredDate for delivery date of the order and the status for status of the order. We store status as enum type. It resembles that enum in C# but we can't specify number.

We create a mapping type variable named orders in order to store the order information by id and to query the order by id. It resembles that Dictionary in C#.

Every account or contract on the blockchain has a 160-bit address. We must save the address of the account that created the contract so that only this account can add orders. So only the marketplace can add orders. We access the account address from msg.sender. We get this information from the constructor function and store it in the owner variable. In Solidity, the address data type is used for address information.

We add a function named add to get the order information. We store this information in the orders variable. We need an authorization. So only the contract owner should be able to call this function. For this, we create a modifier named onlyOwner. In this modifier, we compare the account address that calls the function with the account address that creates the contract. If it doesn't match, we throw "The sender isn't authorized" error. Finally, we add this modifier to the add function. We can add many modifiers here. These modifiers are run orderly and the function is run if there is no error.

We add a function named get to return the details about the order. This function takes the order id as parameter. We check the order and if there is no order, we throw an error.

In Solidity, two keywords, memory and storage, are used for reference types. Smart Contracts consume RAM each time a function is called. Variables marked with memory are cleared when the function completes its work. Variables marked with storage live as long as the contract lives.

Since smart contracts cannot be edited, we need to test contracts before publishing. That's why TDD is important. We write tests for the add function in orders.js under the test folder.

When the test starts running, Ganache defines 10 accounts. We can access these accounts from accounts parameter on line 3.

There are three units in the Ethereum network namely Ether, Gwei and Wei. Although Ether is the largest unit, all transactions are made with the smallest unit Wei. In the 8th line, so we convert 2.5 Ether to Wei.

The contract is always created with the first account. In the 16th line, we give the second account to the from parameter so that we expect an error to occur.

We run the Ganache application and click the QuickStart Ethereum button to activate the local blockchain network.

Ganache

We run the test with the following command line.

truffle test ./test/orders.js

The Test Result of The Add Function

Account Status After The Test of Add Function

Developing Order Paying Function

We create a struct named Payment to store the information that we have received the payment. We add id for the id of the order, buyerAddress for the address of the paying account, paidDate for the payment date, and refundedDate for the refund date. We mark the buyerAddress with payable so that we can refund to that address. We create a variable named payments.

Let's publish an event when the payment is made. We create an event named OrderPaid. The event arguments are id, paidAddress, paidAmount and date.

We add a function named pay to receive the payment. This function takes the order id as a parameter. The difference of this function from the add function is that it can receive payment. That's why we add the payable keyword. We check the order. We throw an error if it has already been paid. Then we compare the price of the order with the payment amount and throw an error if they don't match. We access the payment amount from msg.value.

We update the status of the order to Paid. We store the address of the paying account and the payment date. When payment is made with this function, the amount is stored in the contract. Finally, we publish the OrderPaid event.

block.timestamp returns the time of the current block in milliseconds.
msg.value returns the amount entered when the function was executed.
msg.sender returns the address of the account running the function.

We write tests for the pay function in orders.js under the test folder.

Because the pay method is marked as payable, it can receive payment. In the 16th and 27th lines, we enter 200 Wei to the value parameter, we expect an error because the order amount (2.5 Ether) doesn't match 200 Wei.

We published an event when the payment was successful. In the 37th line, we check whether the event is published or not.

We run the test with the following command line.

truffle test ./test/orders.js

The Test Result of The Pay Function

Account Status After The Test of Pay Function

Developing Order Delivering Function

Let's publish an event when the order is delivered. We create an event named OrderDelivered. The event arguments are id and date.

We add a function named deliver to mark the order as delivered. This function takes the order id as a parameter. We also authorize this function for the marketplace. Just like the add function, we add the onlyOwner modifier to this function.

We check the order. We also throw an error if it has already been delivered.

We update the status of the order to Delivered. We add the delivery date of the order. Finally, we publish the OrderDelivered event.

We write tests for the deliver function in orders.js under the test folder.

We run the test with the following command line.

truffle test ./test/orders.js

The Test Result of The Deliver Function

Account Status After The Test of Deliver Function

Developing Order Refunding Function

Let's publish an event when the refund is made. We create an event named OrderRefunded. The event arguments are id and date.

We add a function named refund for process of returning. This function takes the order id as a parameter. We check the order. We throw an error if the order hasn't been paid for. We throw an error if the order has been delivered. We throw an error if a refund has been made. If the account that makes the payment and the account that calls the function are not the same, we throw an error. We throw an error if the promised delivery date hasn't yet been exceeded.

We make the refund to the buyer. We update the status of the order to Refunded. We add the refund date. Finally, we publish the OrderRefunded event

In order to test whether the promised delivery date has been exceeded, we need to mock block.timestamp . For this, we add a function named getTime and return the current date from this function.

We add a contract named MockedOrders.sol under the contracts folder. We extend this contract from the Orders contract. We add a function named mockTimestamp. This function takes a timestamp as a parameter and stores it. The getTime function returns this timestamp. We will do our test on this contract.

We edit the 2_deploy_contracts.js migration under the migrations folder.

We write tests for the refund function in orders.js under the test folder.

Solidity has the concepts of Gas, Gas Price and Gas Cost. Each function executed in the contract has a certain cost. Gas is calculated automatically according to the size of the contract and the running code. For example, 1000 Gas is calculated for the pay method. Miners are needed to confirm the transactions made in the contract and create a new block. This is where Gas Price comes into play. Miners work with the Gas * GasPrice calculation. In other words, the higher the Gas Price, the faster the transactions are approved :) The result of this multiplication is called Gas Cost.

In the 69th line, we get Gas Price which default value in Ganache is 20000000000 Gwei.

In the 82th line, we get the balance of the third account. Default 100 Ether in Ganache.

In the 85th to 88th lines, we calculate the Gas Cost for the pay and refund functions. The Gas Price is multiplied by the amount of Gas used.

In the 92th line, we again take the balance of the second account.

In the 93th to 94th lines, we add the Gas Costs that we calculated above to the final balance.

In the 96th line, we compare the balances to check if the refund has been successfully made.

We run the test with the following command line.

truffle test ./test/orders.js

The Test Result of The Refund Function

Account Status After The Test of Refund Function

We have developed the contract. In the next post we will look at how to use it in web application.

You can access the sample application on Github.

ahmetkucukoglu / order-blockchain-app

Blockchain Uygulama Geliştirme

1.Bölüm Kontrat Oluşturma : ahmetkucukoglu.com/blockchain-uygulama-gelistirme-1-kontrat-olusturma

2.Bölüm UI İle Etkileşim : ahmetkucukoglu.com/blockchain-uygulama-gelistirme-2-ui-ile-etkilesim

3.Bölüm Kontratı Yayınlama ve API Hazırlama : ahmetkucukoglu.com/blockchain-uygulama-gelistirme-3-kontrati-yayinlama-ve-api-hazirlama

View on GitHub

Good luck.

Bootstrap Filter Plugin

Ahmet Küçükoğlu — Mon, 16 Nov 2020 10:04:30 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/bootstrap-filter-plugin-en/

Instead of the form elements being displayed side by side, it allows the form element to be displayed with the user action.

Basic Form Elements

This use is recommended if the filter contains classic form elements (text, radio, checkbox, select-one or select-multiple). Click for sample page.

HTML

The "data-filter" feature is added to the buttons. The name of the filter is given as a value.

<div class="btn btn-sm btn-outline-secondary position-relative" data-filter-button="monthOfBirth"></div>

The "data-filter-container" feature is added to the containers that will be opened when the buttons are clicked. The name of the filter is given as a value.

<div class="collapse bg-white border rounded pr-1" data-filter-container="monthOfBirth"></div>

The container may include the form elements of text, radio, checkbox, select-one or select-multiple.

<div class="p-3">
    <div class="form-group mb-0">
        <label for="monthOfBirthSelect">Month of Birth</label>
        <select class="form-control" name="monthOfBirthSelect" id="monthOfBirthSelect">
            <option value="">Please select</option>
            <option value="1">January</option>
            <option value="2">February</option>
            <option value="3">March</option>
            <option value="4">April</option>
            <option value="5">May</option>
        </select>
    </div>
</div>

JS

JSON object is created that contains the filter's name, label, default text and default value.

The JSON key must be the same with the "data-filter value" which is added to HTML.

var filters = {
    monthOfBirth: {
        name: 'monthOfBirthSelect',
        label: 'Month of Birth',
        defaultText: 'Please select',
        defaultValue: null
    }
};

filterPlugin.initialize(filters);


name	The name of the form element in the container is indicated.
label	It is used to show the name of the filter in the filter button.
defaultText	It is used to show the default text in the filter button if there is no selection in the form element.
defaultValue	It specifies the value that will be selected in the form element when the reset button or the clear filter button is clicked.

Custom Form Elements

This use is recommended if the filter contains more than one form element or different components. Click for sample page

JS

Three methods are defined for each filter.

The method that returns the text of the filter is defined as "getText".

The method that returns the value of the filter is defined as "getValue".

The method to be called to clear the filter is defined as "clearValue".

The JSON key must be the same as the data-filter value added in HTML.

var filters = {
    birthDate: {
        getText: () => {

            var month = document.getElementById('monthOfBirthSelect');
            var year = document.getElementById('yearOfBirthSelect');

            if (month.selectedIndex == 0 && year.selectedIndex == 0)
                return 'Birth date: Please select';

            var text = '';

            if (month.selectedIndex > 0)
                text += month.options[month.selectedIndex].text;

            if (year.selectedIndex > 0)
                text += ' ' + year.options[year.selectedIndex].text;

            return 'Birth date: ' + text;
        },
        getValue: () => {

            var month = document.getElementById('monthOfBirthSelect');
            var year = document.getElementById('yearOfBirthSelect');

            if (month.selectedIndex == 0 && year.selectedIndex == 0)
                return null;

            var value = {};

            if (month.selectedIndex > 0)
                value['month'] = month.options[month.selectedIndex].value;

            if (year.selectedIndex > 0)
                value['year'] = year.options[year.selectedIndex].value;

            return value;
        },
        clearValue: () => {

            document.getElementById('monthOfBirthSelect').selectedIndex = null;
            document.getElementById('yearOfBirthSelect').selectedIndex = null;
        }
    }
};

filterPlugin.initialize(filters);

Handle Events

This use is recommended if an event triggering is requested when the apply or reset buttons are clicked. Click for sample page

JS

Two methods are defined.

The method to be called when the reset button is clicked is defined as "clickedFilterResetButton".

The method to be called when the Apply button is clicked is defined as "clickedFilterApplyButton".

Filter values are passed as parameters (payload) to "clickedFilterApplyButton" method in JSON format.

var filters = {
    monthOfBirth: {
        name: 'monthOfBirthSelect',
        label: 'Month',
        defaultText: 'Please select',
        defaultValue: null
    }
};

var callbacks = {
    clickedFilterResetButton: () => {

        console.log('Clicked reset button');
    },
    clickedFilterApplyButton: (payload) => {

        console.log('Clicked apply button', payload);
    }
};

filterPlugin.initialize(filters, callbacks);

You can access source code and samples from Github.

ASP.NET Core Feature Management

Ahmet Küçükoğlu — Sun, 15 Nov 2020 12:42:59 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/asp-net-core-feature-management-en/

Feature Management provides feature management in .NET Core applications. It allows the management and querying of the active / passive status of the features of the application. For example, you can ensure that a feature you have just developed is active in a certain date range. To give another example, you can ensure that a feature you have developed is active with a certain percentage. Like A/B testing.

Let's test it by practicing. First, we need to install the package below.

dotnet add package Microsoft.FeatureManagement.AspNetCore -version 2.0.0

Let's define our features in enum.

Let's activate the Feature Management in the Startup.

Because Feature Management reads features from IConfiguration, we can define features in environment variable, commandline argument or appsettings. Or, if you have a custom provider, you can define your features in the related source. We will use appsettings in our example. Let's define the feature named FeatureA.

We need to inject the "IFeatureManager" interface to query if the features are activated. By using the "IsEnabledAsync" method of this interface, we can query as follows if the "FeatureA" feature is activated.

TimeWindowFilter

If we want the feature to be active in a certain date range, we can use the internal "TimeWindowFilter". For this, we need to activate this filter in the Startup.

Let's define the feature named FeatureB in appsettings.

UTC Date is used as the date format. If you only give the Start parameter, the feature will be active from this date. If you only give the End parameter, the feature will be active until this date.

Again, using the "IsEnabledAsync" method of the "IFeatureManager" interface, we can query if the "FeatureB" feature gets activated as follows.

PercentageFilter

If we want the feature to be active with a certain percentage, we can use the internal "PercentageFilter". For this, we need to activate this filter in the Startup.

Let's define the feature named FeatureC in appsettings.

This feature will be active in 80% of the requests.

Again, using the "IsEnabledAsync" method of the "IFeatureManager" interface, we can query if the "FeatureC" feature gets activated as follows.

Custom Filter

Apart from these filters, you can also write your own filters. For example, a filter can be written as below for the feature to be active on a mobile device.

Let's define the feature named FeatureD in appsettings.

Let's create "MobileFilterSettings" class to access filter parameters.

Let's write our special filter named "MobileFilter" as follows.

We need to activate this filter in the Startup.

Again, using the "IsEnabledAsync" method of the "IFeatureManager" interface, we can query if "FeatureD" feature gets activated as follows.

Mvc Action Filter Attribute

Instead of using the IFeatureManager interface, you can set the activation of the attribute based Controller or Action, thanks to ActionFilter.

Razor Tag Helper

You can use the feature razor tag if you want to display the content in the View according to the activation status of the feature.

Middleware

If you want to add middleware according to the activation status of the feature, you can use the UseMiddlewareForFeature method.

You can access the sample project from Github.

Good luck.

Event Sourcing with ASP.NET Core - 02 Messaging

Ahmet Küçükoğlu — Thu, 22 Oct 2020 23:15:20 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/event-sourcing-with-aspnet-core-02-messaging/

1. Introduction

Since this article is second part of the article below, I recommend you to read the following article before starting. We will continue with the sample project in the article below.

https://www.ahmetkucukoglu.com/en/event-sourcing-with-asp-net-core-01-store/

In the previous article, we made the example of Kanban Board. By creating a RESTful API, we wrote the create, assign, move and complete endpoints. We recorded the requests coming to these endpoints as an event in the Event Store. So we focused on the store part of the Event Store. In this article, we will focus on with the messaging part.

We will include the following endpoint in our RESTful API endpoints.

[GET] api/tasks/{id}

2. Preparing Couchbase

Since the events are not queryable, the events must be stored as nosql documents or sql row to represent the whole. For example; We can't get tasks which their states appear as "Done, from the Event Store. For this reason, we need to keep the events as a whole in a queryable database. In this example, we will keep the tasks on the couchbase.

We run Couchbase in Docker with the command line below.

docker run -d --name couchbase -p 8091-8094:8091-8094 -p 11210:11210 couchbase

When the Couchbase has been run, you should enter the panel at the address below and create a cluster.

http://localhost:8091/

Let's create two buckets named "checkpoints" and "tasks" in Couchbase with the curl command lines below. Remember to replace the -u parameter with the information you have provided when creating the cluster.

curl -X POST -u [admin]:[password] http://localhost:8091/pools/default/buckets
  -d name=checkpoints -d ramQuotaMB=100 -d authType=none -d bucketType=couchbase

curl -X POST -u [admin]:[password] http://localhost:8091/pools/default/buckets
  -d name=tasks -d ramQuotaMB=100 -d authType=none -d bucketType=couchbase

3. Adapting Couchbase to API

We install the packages "CouchbaseNetClient" and "Couchbase.Extensions.DependencyInjection" to the API project with the following command lines.

dotnet add package CouchbaseNetClient -v 2.7.16
dotnet add package Couchbase.Extensions.DependencyInjection -v 2.0.2

We add Couchbase connection information to appsettings.json as follows.

In the Startup.cs file, we configure the Couchbase.

Each event has a position value in the Event Store. After recording the events read from the Event Store to Couchbase, we must keep the position value of the last event read. We will store this position value in the CheckpointDocument type.

Let's add a class named "CheckpointDocument.cs" in the Infrastructure folder. Let's paste the code below into the class.

We write the repository class to read the last position value that we have written to Couchbase or to record the position value.

Let's add a class named "CheckpointRepository.cs" in the Infrastructure folder. Let's paste the code below into the class.

We will store task information in TaskDocument type. Let's add a class named "TaskDocument.cs" in the Infrastructure folder. Let's paste the code below into the class.

We write the repository class to write the events read from the Event Store to Couchbase and to query the tasks from Couchbase.

Let's add a class named "TaskRepository.cs" in the Infrastructure folder. Let's paste the code below into the class.

In the save method, we add the relevant task according to the type of the event recorded in the EventStore and update the related information of the task.

In the Get method, we query the tasks from Couchbase by id .

Let's add CheckpointRepository and TaskRepository classes to DI Container in Startup.cs file as below.

4. Preparing Consumer

We need the consumer to listen to the events from the Event Store. For this reason, we will use the Hosted Service in ASP.NET Core.

Let's add a folder named "HostedServices" to the project and add a class named "TaskHostedService.cs" into it. Let's paste the code below into the class.

In the line 33, we query the position value of the last event recorded to Couchbase.

In the line 42, we subscribe to the Event Store and start listening to the events. If lastCheckpoint value is not given, all events are read repeatedly whenever we run the project. With lastCheckpoint, we state that we will read the events coming after the last position value we have read.

In the line 47, we skip in order not to take action on the Event Store's own events.

In the line 52, we find the type of the event which is read.

In the line 53, we deserialize the event according to its type.

In the line 55, we skip in order not to take action for events other than these events.

In the line 58, we record the task to Couchbase.

In the line 60, we record the position value of the event read from the Event Store to Couchbase.

5. Preparing API Endpoint

We add the endpoint where we will query the task to TasksController as follows.

Everything is ready now. We can run the API project. We can query the task we have added at the end of the previous article with the curl command line below.

curl https://localhost:44361/api/tasks/3a7daba9-872c-4f4d-8d6f-e9700d78c4f5

You can access the final version of the project from Github.

Good luck.

Event Sourcing with ASP.NET Core – 01 Store

Ahmet Küçükoğlu — Mon, 25 May 2020 14:54:41 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/event-sourcing-with-asp-net-core-01-store/

1. Introduction

I recommend you to read the article below before applying this example tutorial.

https://www.ahmetkucukoglu.com/en/what-is-event-sourcing/

In the article I have mentioned above, I had formed a sentence as follows.

There is a technology called “Event Store” in the .NET world for Event Sourcing. This technology offers solutions for “Aggregate” and “Projection”. In other words, in addition to providing the store where we can record the events, it also provides the “Messaging” and “Projection” services, which are necessary for us to record in “Query” databases.

In this article, we will deal with the store section of the Event Store. In other words, we will deal with the database feature where we can save events. In the next article, we will deal with the messaging part.

As an sample application, I chose the classic Kanban Board sample.

Our RESTful API endpoints will be as follows.

[POST] api/tasks/{id}/create

[PATCH] api/tasks/{id}/assign

[PATCH] api/tasks/{id}/move

[PATCH] api/tasks/{id}/complete

Our events will be as follows.

CreatedTask : When Task is created, an event with this name will be created.

AssignedTask : An event with this name will be created when Task is assigned to someone.

MovedTask : When Task is moved to a section (In Progress, Done), an event with this name will be created.

CompletedTask : When Task is completed, an event with this name will be created.

2. Installing the Event Store

We run the Event Store in Docker with the following command line.

docker run -d --name eventstore -p 2113:2113 -p 1113:1113 eventstore/eventstore

When the Event Store run, you can enter the panel from the address below. The default username is "admin" and the password is "changeit".

http://localhost:2113/

3. Creating the API Project

We create an ASP.NET Core Web API Application project named "EventSourcingTaskApp". We install the "EventStore.Client" package to the project with the following command line.

dotnet add package EventStore.Client -v 5.0.6

We add Event Store connection information to appsettings.json as follows.

In the Startup.cs file, we connect the Event Store and add it to the DI Container.

4. Aggregate Base Class and Aggregate Repository

Let's add two folders named "Core" and "Infrastructure" to the project. We will add Task-related entities, events and exceptions to the Core folder. For the Infrastructure folder, we will add our repository class that will connect the Event Store.

4.1. Aggregate Base Class

Let's add a folder named "Framework" inside the Core folder and add a class named "Aggregate.cs" into it. Let's paste the code below into the class.

This is a standard class. Aggregates are derived from this base class. In our example, Task is an aggregate and this will be derived from the base class.

In the line 9, we define the variable where aggregate events will be stored.

In the line 11 we state that aggregate will have an Id.

In the line 12 we state that the default version of aggregate is "-1".

In the line 16, we write the method that will add events to the variable defined on the line 9.

In the line 23, we write the method that will apply the events to aggregate. The final version of aggregate will be created by running this method for each event read from the Event Store.

In the line 33, we write the method that returns the events on aggregate. While sending events to the Event Store, this method will be run and events will be received.

4.2. Aggregate Repository

Let's add a class named "AggregateRepository.cs" inside the Infrastructure folder. Let's paste the code below into the class.

This is also a standard class. We use this repository when sending events to the Event Store or receiving events from the Event Store.

4.2.1. Sending an Event (Append Events to Stream)

In the line 22, we take the events on aggregate and map them to the EventData class. Event Store stores events in the EventData type.

As the first parameter, it takes for the event's id.

As the second parameter, it takes for the name of the event. Example event name; CreatedTask, AssignedTask etc.

As the third parameter, it takes whether the event data is in json type.

As the fourth parameter, it takes the event data. Since it takes in byte array type, serialize and encoding processes are performed.

As the fifth parameter, it takes the metadata. This parameter can be passed as null but we pass the type of the event class as "fullname" so that we can use this class type when deserializing the events. Example fullname; EventSourcingTaskApp.Core.Events.CreatedTask.

In the line 36, we set the stream name. Event aggregates in the Event Store is called stream. So aggregate is expressed as a stream in Event Store. Example stream name; Task-518e7c15-36ac-4edb-8fa5-931fb8ffa3a5.

In the 38th line, events are recorded in the Event Store.

4.2.2. Reading an Event (Read Events from Stream)

In the line 47, we set the stream name.

In line 53, events are received from the Event Store in order according to the version numbers in the loop.

In the 58th line, the load method of aggregate is called and the events are applied to aggregate and the final form of aggregate is created.

Let's add AggregateRepository class to DI Container in the Startup.cs file as below.

5. Defining Task and Use Cases

In this part, we will start creating events, exceptions, and use cases related to Task.

5.1. Defining Task

Let's add an aggregate class named "Task.cs" in the Core folder. Let's paste the code below into the class.

Task will have the Title, Section, AssignedTo and IsCompleted.

Again, let's add a class named "BoardSections.cs" into Core. Let's paste the code below into the class.

5.2. Defining Exceptions

Let's add a folder named "Exceptions" inside the Core folder and add a class named "TaskAlreadyCreatedException.cs" in it. Let's paste the code below into the class. If it tries to create a task with the same id information, we will throw this error.

Let's add a class named "TaskCompletedException.cs" in the Exceptions folder inside the Core folder. Let's paste the code below into the class. If any process is tried on the completed task, we will throw this error.

Let's add a class named "TaskNotFoundException.cs" inside the Exceptions folder inside the Core folder. Let's paste the code below into the class. If Task is not found, we will throw this error.

Now that we have created the Task aggregate and exceptions, we can start writing use cases related to the Task.

5.3. Create Task

When Task is created; We will keep task id, its title and the information of who created the task, as event data.

Let's add a folder named "Events" inside the Core folder and add an event class named "CreatedTask.cs" into it. Let's paste the code below into the class.

Let's edit the Task.cs class as follows.

With the Create method, we store the CreatedTask event on aggregate, so that we can send these stored events to the Event Store.

In the line 19, we apply the CreatedTask event from the Event Store to aggregate.

5.4. Assign Task

When Task is assigned to someone; We will keep task id , information of who has assigned the task and to whom the task has been assigned as event data.

Let's add an event class named "AssignedTask.cs" to the Events folder inside the Core folder. Let's paste the code below into the class.

Let's edit the Task.cs class as follows.

With the Assign method, we store the AssignedTask event on aggregate so that we can send these stored events to the Event Store.

In the line 20, we apply AssignedTask event received from Event Store to aggregate.

5.5. Move Task

When Task is moved to "In Progress" or "Done" section; We will keep the task id, information of who has moved the task and to which part the task has been moved, as event data

Let's add an event class named "MovedTask.cs" to the Events folder inside the Core folder. Let's paste the code below into the class.

Let's edit the Task.cs class as follows.

We store the MovedTask event on aggregate with the Move method, so that we can send these stored events to the Event Store.

In the line 21, we apply the MovedTask event received from the Event Store to aggregate.

5.6. Complete Task

When Task is completed; We will keep task id, who has completed the task as event data.

Let's add an event class named "CompletedTask.cs" to the Events folder inside the Core folder. Let's paste the code below into the class.

Let's edit the Task.cs class as follows.

With the Complete method, we store the CompletedTask event on aggregate so that we can send these stored events to the Event Store.

In the line 22, we apply the CompletedTask event received from the Event Store to aggregate.

6. Preparing API Endpoints

Let's create a controller named "TasksController" and paste the code below.

As you can see in the actions, firstly, events are taken from the Event Store with the AggregateRepository's Load method and aggregate is created. Then, new events are stored in aggregate with the use case methods on aggregate. With the AggregateRepository’s Save method, these stored events are sent to the Event Store.

Let's run the API and send requests to the API.

Let's create a task with the curl command line below.

curl -d "title=Event Store kurulacak" -H "Content-Type: application/x-www-form-urlencoded" -X POST https://localhost:44361/api/tasks/3a7daba9-872c-4f4d-8d6f-e9700d78c4f5/create

Let's assign the task to someone with the curl command line below.

curl -d "assignedTo=Aziz CETİN" -H "Content-Type: application/x-www-form-urlencoded" -X PATCH https://localhost:44361/api/tasks/3a7daba9-872c-4f4d-8d6f-e9700d78c4f5/assign

Let's pull the task to In-Progress with the curl command line below.

curl -d "section=2" -H "Content-Type: application/x-www-form-urlencoded" -X PATCH https://localhost:44361/api/tasks/3a7daba9-872c-4f4d-8d6f-e9700d78c4f5/move

Let's pull the task to Done with the curl command line below.

curl -d "section=3" -H "Content-Type: application/x-www-form-urlencoded" -X PATCH https://localhost:44361/api/tasks/3a7daba9-872c-4f4d-8d6f-e9700d78c4f5/move

Let's complete the task with the curl command line below.

curl -d -H "Content-Type: application/x-www-form-urlencoded" -X PATCH https://localhost:44361/api/tasks/3a7daba9-872c-4f4d-8d6f-e9700d78c4f5/complete

You can check the stream by entering the Event Store panel.

http://localhost:2113/web/index.html#/streams/Task-3a7daba9-872c-4f4d-8d6f-e9700d78c4f5

You can access the final version of the project from Github.

Good luck.

Couchbase GeoSearch with ASP.NET Core

Ahmet Küçükoğlu — Wed, 20 May 2020 04:26:53 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/couchbase-geosearch-with-asp-net-core/

The subject of this article will be about how to do "GeoSearch" by using Couchbase.

1. Installing the Couchbase

For this purpose, we create Couchbase cluster with the docker by running the command below.

docker run -d --name couchbase -p 8091-8094:8091-8094 -p 11210:11210 couchbase

When Couchbase is up, it will start broadcasting at the address below.

http://localhost:8091/

We create the cluster by clicking the "Setup New Cluster" button. We define the password as "123456".

You can make adjustment according to your current memory status. You can turn off "Analytics". We complete the cluster installation by clicking the "Save & Finish" button.

We go to "Buckets" from the left menu, click the "Add Bucket" button at the top of the right corner and create a bucket called "meetup".

http://localhost:8091/ui/index.html#!/buckets

2. Creating the API

As an example, we will do the events section in the Meetup application that we all use. We will define events through the API. We will record the locations of these events. We will then query a list of events near by the current location.

We create an ASP.NET Core API project. We install the nuget packages below.

dotnet add package CouchbaseNetClient
dotnet add package Couchbase.Extensions.DependencyInjection

We add Couchbase connection information to the appsettings.json file.

We define Couchbase in the Startup.cs file.

2.1. Creating an CreateEvent Endpoint

In the project, we create a folder named "Models" and add a class named "EventDocument" into it. This class will be the model of the document that we will add to the bucket named "events" in Couchbase.

Likewise, we add a class named "CreateEventRequest" under the "Models" folder. It will be the request model of endpoint, which will allow us to record this event.

We add a controller named "EventsController".

Now we can add events to Couchbase. Let's run the application and make requests from Postman as follows.

When we look at the documents of events bucket in Couchbase, we will see that it has been added.

http://localhost:8091/ui/index.html#!/doc_editor?bucket=events

2.2. Creating The Index That Allows Searching According To Location

In Couchbase, we come to "Search" from the left menu and click the "Add Index" button in the upper right corner.

http://localhost:8091/ui/index.html#!/fts_new/?indexType=fulltext-index&sourceType=couchbase

We write "eventsgeoindex" in the Name field.

We select "events" from the Bucket field.

Using the "+ insert child field" on the right side of the mapping named "default" under "Type Mappings", we add mappings as follows.

We add mapping to save the "Subject" information in the event to "eventsgeoindex".

We add mapping to save the "Address" information in the event to "eventsgeoindex".

We add mapping to save the "Date" information in the event to "eventsgeoindex".

We add mapping to search according to the "Location" information in the event.

The final situation will be as in the image below.

We create the index by clicking the "Create Index" button. When "Indexing progress" is 100%, it means that indexing has been finished.

http://localhost:8091/ui/index.html#!/fts_list?open=eventsgeoindex

2.3. Creating The Endpoint GetNearbyEvents

We add a class named "GetUpcomingEventsRequest" under the "Models" folder. This will be the request model of endpoint, which will return events near by our location.

Likewise, we add a class named "GetUpcomingEventsResponse" under the "Models" file. This will be the endpoint's response model, which returns events that are near by our location.

We add a controller named "UpcomingEventsController".

In the line 30, we indicate that the search will be made on the basis of km.

In the line 40, we indicate the name of the "search index" that we will create in Couchbase.

Now we can list events near by our location. Let's run the application and make requests from Postman as follows. By typing 1 and 2 to "Radius", you can see the events near 1 or 2km.

You can access the final version of the project from Github.

Good luck.

What is Event Sourcing?

Ahmet Küçükoğlu — Sun, 12 Apr 2020 09:17:08 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/what-is-event-sourcing/

Previously, our services would be as follows.

This service consisted of hundreds of lines of code. To get rid of this confusion, we split our services into two parts as "Command" and "Query" with CQRS. We performed our CRUD operations with "Command" services and our query operations with "Query" services. Two separate services appeared as follows.

In this way, we achieved a little readability, but god classes were still emerging. "UseCase" s have entered our lives with Mediator. This time we started creating separate handlers for each "UseCase". 13 separate handlers appeared as follows.

With this approach, each handler just did its own job. The lines of code decreased further, readability increased even more. This gave us easy maintainability.

This time we worked on performance. Each emerging database technology offered different features. Some of them was very good for fast writing and some of them was good for fast reading. Since we have seperated everything… We chose database technologies with good writing speed for “Command” operations and database technologies with good reading speed for “Query” operations. There was a problem. We had to synchronize data in two different database technologies.

At this point, "Messaging" technologies came to our rescue. After writing the data to the database in the command handler, we sent the same data to the queues as a message. We wrote a consumer application that listens to messages in the queue. We transferred the messages that came with the consumers to the database which we had selected for “Query” operations. In this way, we achieved synchronization between two different database technologies.

Then we realized that not everything was CRUD. It wasn't all about adding, updating and deleting data. What were we updating?

Let's consider the Meetup application.

For example, we define an event. This is an insert.

After defining the event, we can update the content of the event. This is an update.

Those who plan to attend the event state that they will go to the event. Is this an update for the event? If it is not update, what is it?

We mark the status of the event as "Stopped" an hour before the event has started.

After the event has been finished, we mark the status of the event as "Completed". So is this an update?

As the owner of the event, we upload photos to the event. Is this also an update?

Participants enter comments on the event. Do you think that it is an Update?

As you can see, everything could not be thought of as Insert, Update and Delete. In fact, everything was an event as the following.

MeetupRegistered
MeetupJoined
MeetupStopped
MeetupCompleted
MeetupAddedPhoto
MeetupAddedComment

If everything was an event, there was no need to keep our data in rows in related tables as we do in the classical method. Then we created a table as in the image below and added each event to this table as a line.

When we look, we see that the event we have defined is just a join of these rows. Each event has its version number. When we join it from bottom to top, the result is as follows.

The event that we have defined represents "Aggregate" as a whole with its information, photos and comments. That's why AggregateId information is available.

We said that in the "Command" operations we save our data as an event in a database as above. But since we cannot query these rows, we also need to write these events to the database we choose for "Query". For example when we want to show "incomplete events" in the meetup application, we cannot make an query in the events above.

As I have mentioned in the previous paragraphs, after writing the event information to the database as an event in "Command" operations, we should send this information to a "messaging" queue. We should write a consumer listening to this queue. Through this consumer, we should listen to the "MeetupRegistered" events and write the necessary information and the event information to the "Query" database. In the same way, we should also listen to the "MeetupCompleted" events and delete the event information from the "Query" database.

As you can see above, events from the API in the "Command" area are written to the event store. These events are also sent to the queue. In the "Projection" area, the messages in the queue is written by being processed to the query store. Queries are made in the "Query" area, and results are returned from the API.

This is Event Sourcing. It has unique terms like Aggregate, Projection, Snapshot etc. So, what technologies can we choose for "Command", "Messaging" and "Query"?

"MSSQL" can be used for events. "RabbitMQ" can be used for "Messaging". Technologies such as "RavenDB, ElasticSearch, Couchbase" can be used for "Query". But for Event Sourcing, there is a technology called "Event Store" in the .NET world. This technology offers solutions for "Aggregate" and "Projection". In other words, in addition to providing the store where we can record the events, it also provides the "Messaging" and "Projection" services, which are necessary for us to record in “Query” databases.

Event Store

I have been adding practical informations to my blog rather than theoretical one. So I have prefered to write Tutorial style articles. But this time, I wanted to give first some theoretical informations for Event Sourcing. I intend to come up with a video that we will put the theory into practice soon. In that video, I will develop a sample application without going into definitions too much. For this reason, I wanted to establish the basis for that video with this theoretical article.

Good luck.

Publishing Docker Image as Serverless on GCP

Ahmet Küçükoğlu — Wed, 26 Feb 2020 06:57:01 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/publishing-docker-image-as-serverless-on-gcp/

1. Introduction

With the serverless approach, the problem about the server that the software developer confronts with has disappeared. Now we write our code and call the provider to run it. We are not dealing with server configurations, scalings etc. And pay as you go. This is a great opportunity.

AWS's Lambda, GCP's Cloud Function, Azure's Azure Function etc., these services provide us this service. But this time, different kind of problem appears. These services do not support all languages. Even if it supports the language we want, it may not support the version we want. For example, AWS Lambda does not support the upper versions of .NET Core 2.1 yet. This is a problem. Nobody can restrict us :)

The Cloud Run service that Google built on Knative completely solves this problem. We give Cloud Run a docker image and Cloud Run runs our serverless service from this image. So whether we develop in PHP or .NET Core, it doesn't matter. There is no limit as long as we can make dockerize. This is awesome.

2. Preparing the Development Environment

In this part we will arrange our development environment. You can apply exactly the second part of my previous article, Publishing the ASP.NET Core Application in GCP Kubernetes.

3. Preparing GCP

3.1. Creating the Project

Before we can use Google Cloud services, we need to create a project. You need to run the following command from the terminal. You can update PROJECT_ID and PROJECT_NAME fields according to your.

gcloud projects create [PROJECT_ID] --name=[PROJECT_NAME]
gcloud projects create my-cloud-run-api-project-1 --name="My Cloud Run API Project"

You can run the command below to see the project created.

gcloud projects list

3.2. Choosing the Project

We need to define the project that we have created on google cloud config on our local machine. You need to run the following command from the terminal. Remember to update the PROJECT_ID field according to you.

gcloud config set project [PROJECT_ID]
gcloud config set project my-cloud-run-api-project-1

3.3. Activating the Billing

Before we publish the application, we need to link an billing account to the project that we have created. To learn your Account Id, you need to run the following command from the terminal.

gcloud beta billing accounts list

You need to run the following command to install the alpha commands of the Google SDK.

gcloud components install alpha

Once the process is complete, you need to run the following command to link the Account Id to your project. Remember to update the PROJECT_ID and ACCOUNT_ID fields according to you.

gcloud alpha billing projects link [PROJECT_ID] --billing-account [ACCOUNT_ID]

You can run the command below to see that the bill has been successfully linked to your project. Remember to update the ACCOUNT_ID field according to you.

gcloud beta billing projects list --billing-account=[ACCOUNT_ID]

3.4. Activating the Cloud Run API

It is required to activate Cloud Run API from Google Cloud APIs. You need to run the following command from the terminal.

gcloud services enable run.googleapis.com

3.5. Activating the Container Registry API

Container Registry API needs to be activated from Google Cloud APIs. You need to run the following command from the terminal.

gcloud services enable containerregistry.googleapis.com

4. Publishing the Application

I created a simple API. You can pull this API from Github with the command below.

git clone https://github.com/ahmetkucukoglu/aspnetcore-gcloud-run-sample.git

There is one thing we need to pay attention. It is that when Cloud Run running a service from the docker image, it set the port on its own. It writes this port to environment variable. It wants us to run the application that we have written in a way for this port to listen. If we don't, Cloud Run stops service if it doesn't receive a response when it does health check.

You can see how it is set in lines 11 and 12.

4.1. Creating the Image and Sending the Image to GCR

The project includes Dockerfile. We should create an image in the docker with the following command. You can set the IMAGE_ID field according to you.

docker build -t [IMAGE_ID] -f CloudRunSampleAPI/Dockerfile .
docker build -t my-cloud-run-api-image:1.0 -f CloudRunSampleAPI/Dockerfile .

Let's first create a tag to send this image we have created to GCR. Remember to update the LOCAL_IMAGE_ID, PROJECT_ID and IMAGE_ID fields according to you.

docker tag [LOCAL_IMAGE_ID] gcr.io/[PROJECT_ID]/[IMAGE_ID]
docker tag my-cloud-run-api-image:1.0 gcr.io/my-cloud-run-api-project-1/my-cloud-run-api-image:1.0

Now, let's send the image to GCR using this tag.

docker push gcr.io/[PROJECT_ID]/[IMAGE_ID]
docker push gcr.io/my-cloud-run-api-project-1/my-cloud-run-api-image:1.0

To see the image sent to GCR, you can run the command below from the terminal.

gcloud container images list

4.2. Deploying the Application

We have to deploy our application using the image we created above. You need to run the following command from the terminal. Remember to update the SERVICE_NAME and IMAGE_ID fields according to your own information.

gcloud beta run deploy [SERVICE_NAME] --image [IMAGE_ID] --platform managed --region europe-west1 --update-env-vars APIKey=Key123,APISecret=Secret456
gcloud beta run deploy my-cloud-run-api --image gcr.io/my-cloud-run-api-project-1/my-cloud-run-api-image:1.0 --platform managed --region europe-west1 --update-env-vars APIKey=Key123,APISecret=Secret456

We saw the endpoint at the terminal. You can test it now.

You can access the final version of the project from Github.

Good luck.

Developing AWS Serverless Messaging System

Ahmet Küçükoğlu — Tue, 18 Feb 2020 06:03:21 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/developing-aws-serverless-messaging-system/

1. Introduction

This article will be about how to develop the messaging system with the serverless approach. We will use AWS as a cloud provider. We will prefer .NET Core as the language.

In this part, I will set up the scenario and give preliminary information about what will be the result.

Our RESTful API endpoints will be as follows.

[POST] api/comments

[GET] api/comments

We will use the AWS services below.

API Gateway : It will provide the endpoints we need for the RESTful API.

Lambda : It will provide us to write function for the GET and POST requests.

SQS : It will provide a messaging system. In POST request, we will send the comment texts to the queue for later processing.

Comprehend : It will provide machine learning system. We will use it to detect the languages of the comment texts entered.

DynamoDB : It will provide NoSQL solution. In the POST request, we will write the data to the DynamoDB.

S3 : It will provide our code to be versioned and stored after each deployment.

CloudFormation : It will provide us to create and manage automatically all services mentioned above.

When the project is completed, the architecture on AWS will be as follows.

We will get the comment text from the API with the POST method. We will save this comment to DynamoDB and return result. Since it will take a long time to automatically detect the language of the comment text, we will send the comment text to the queue. The lambda function, which listens to the messages in the queue will use the Comprehend service to detect the language. After getting the result, the record in DynamoDB will be updated. We will return the final version of the comment texts with the GET method from the API.

2. Preparing Development Environment

In this part, we will arrange the development environment. We can apply the second part of the article, Developing AWS Serverless RESTful API. We need to include the following policies in the permission section.

AmazonSQSFullAccess
ComprehendFullAccess

3. Preparing API Project

In this part, we will create the project in its simplest form and make deployment. We will not send data to DynamoDB and SQS yet. We will send request to endpoints and get response.

Create the project folder with the command line below.

mkdir serverless-sqs
cd serverless-sqs
code .

We will create two separate projects. One is the API project and the other is the Consumer project, which listens to the messages in the queue. Create an api folder with the following command line from the terminal from VS Code and enter the folder.

mkdir api
cd api

Create the serverless project with the command line below.

serverless create --template aws-csharp --name comments-api

We will see a file named "Handler.cs". This is the entry point of the application. We will need two functions, GET and POST. While this is possible in Nodejs, it is troublesome in .NET Core. Because we will need to edit the output manually and it is something you would not like :). Therefore, we will make arrangements for the ASP.NET Core API to run when a request comes to lambda. In this way, we will direct the requests to actions on the controller with the help of proxy.

Start with cleaning first. Delete the file "Handler.cs". Delete the installed packages with the following command lines.

dotnet remove package Amazon.Lambda.Core
dotnet remove package Amazon.Lambda.Serialization.Json

Change the "hello.zip" in the build.cmd and build.sh files in the project as "api-handler.zip". When we build the project, an artifact named properly shall be created. :) Shortly after, I will show how to use this artifact in serverless.yml.

We need to install some packages because we will run ASP.NET Core API. Install the packages with the following command lines.

dotnet add package Microsoft.AspNetCore.App -v 2.1.4
dotnet add package Amazon.Lambda.AspNetCoreServer
dotnet add package AWSSDK.Extensions.NETCore.Setup

Create a file named "Startup.cs" and paste the following code.

Now, set the entry point of the lambda. Create a file named "LambdaEntryPoint.cs" and paste the code below.

Update the serverless.yml file with the code below.

On the 16th line, specify the entry point of the application as a LambdaEntryPoint file.

On the 19th line, create a proxy endpoint in the API Gateway. In this way, we transfer the requests coming to API Gateway directly to the Controllers matching in the project.

On the 23th line, specify the path of the artifact we need when deploying the application.

Now we can create Controller. Create a folder named "Controllers" and create a file named "CommentsController.cs" in it. Paste the code below.

Create a folder named "Models" and create a file named "CommentsPostRequest.cs" in it. Paste the code below.

Build the code with the command line below.

build

Make our first deployment to AWS with the command line below.

serverless deploy

After the deployment process, it will write 1 endpoint and 1 API Key on the console.

Let's look at what changed in AWS after the process.

A Stack has been created in the Cloud Formation.

Our code has been sent to S3.

API Gateway has been created for the endpoints.

Lambda function has been created.

4. Integrating DynamoDB and SQS into API Project

In this part, we will write the comment text to DynamoDB in the POST request coming from the API and send a message the queue in SQS. In the GET request, we will return the comment texts from DynamoDB.

Install the packages we need with the following command lines.

dotnet add package AWSSDK.DynamoDBv2
dotnet add package AWSSDK.SQS

Update the serverless.yml file as follows.

On the 43th line, define the resource that will be created for DynamoDB.

On the 46th line, name the table.

On the 66th line, export the ARN of DynamoDB that we have created. On the 72th line, export the name of the table. We will use DynamoDB by importing this ARN and the table name we have exported to the consumer project.

On the 11th line, assign the name of the table we have created as environment variable.

On the 23th line, give full access to DynamoDB, which we have created on lambda.

On the 40th line, define the resource that will create the SQS.

On the 43th line, name the queue.

On the 78th line, export the ARN of the SQS we have created. We will use SQS by importing this ARN we have exported to the consumer project.

On the 13th line, assign the name of the table we have created as environment variable.

On the 18th line, give full access to lambda that it to SQS.

Update the Startup.cs file as follows.

On the 12th and 1t3th lines, register DynamoDB and SQS clients to IOC Container.

Create a file named "CommentsGetResponse.cs" under the Models folder and paste the code below.

Create a file named "CommentsQueueRequest.cs" under the Models folder and paste the code below.

Update the CommentsController.cs file with the code below.

On the 39th line, return the comment texts from DynamoDB.

On the 61th line, save the comment text to DynamoDB.

On the 76th line, send a message to the queue we have created in SQS to be processed.

Build the code with the command line below.

build

Make our deployment to AWS with the command line below.

serverless deploy

Make a POST request.

Make a GET request.

Let's look at what changed in AWS after the process.

A table has been created in DynamoDB and the record has been sent to the table with the POST request.

A queue has been created in SQS and a message has been sent to the queue with the POST request.

5. Preparing the Consumer Project

In this part, we will create the consumer project, which will process the messages we have sent to the queue in SQS. Create the consumer folder by running the following command line in the terminal from VS Code and enter it.

mkdir consumer
cd consumer

Create our serverless project with the command line below.

serverless create --template aws-csharp --name comments-consumer

Start with cleaning first again. Delete the file named Handler.cs.

Change the "hello.zip" in the build.cmd and build.sh files in the project as "consumer-handler.zip". When we build the project, an artifact named properly shall be created. Shortly after, I will show how to use this artifact in serverless.yml.

It is necessary to install SQSEvents package to listen to messages in the queue in SQS, and Comprehend package to find the language of the comment text, and DynamoDB package for NoSQL. Install the packages with the following command lines.

dotnet add package Amazon.Lambda.SQSEvents
dotnet add package AWSSDK.Comprehend
dotnet add package AWSSDK.DynamoDBv2

Create a folder named "Models" and create a file named "CommentsQueueRequest.cs" in it and paste the code below.

Create a file named "ConsumerHandler.cs" and paste the code below.

On the 21th line, take the message in SQS. Since we will take the messages one by one in our example, it is enough to take the value in the zero index. For example, if we were going to take the messages ten by ten, we would have to loop them.

On the 32th line, make the language querying of the comment text from the Comprehend service.

On the 48th line, update the line in DynamoDB.

Update the serverless.yml file with the code below.

On the 9th line, import the name of the DynamoDB table that we have exported in api project.

On the 13th line, give full access to the SQS we have created.

On the 18th line, give access to the language querying function in the Comprehend service.

On the 23th and 24th lines, give access to DynamoDB.

On the 35th line, connect the trigger to the queue in SQS.

On the 38th line, define that the messages should be transmitted to lambda one by one.

Build our code with the command line below.

build

Make our deployment with the command line below.

serverless deploy

Let's look at what changed in AWS after the process.

A stack has ben created in Cloud Formation.

Our code has been sent to S3.

Our Lambda function has been created.

In the previous part, we have made a POST request and send a message on the queue in SQS. Now, that message has been processed. When you make a GET request, you will see that the language of the comment text is updated.

You can continue trying by making POST requests.

You can access the final version of the project from Github.

Good luck.

Publishing the ASP.NET Core Application in GCP Kubernetes

Ahmet Küçükoğlu — Tue, 04 Feb 2020 13:30:17 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/publishing-the-asp-net-core-application-in-gcp-kubernetes/

1. Introduction

The subject of this article series is about how to publish API, which we developed with ASP:NET Core, in Kubernetes. We will use GCP(Google Cloud Platform) as a cloud provider.

In this part, i will show the scenario and give preliminary information about what will be the result. Also i will mention briefly some concepts in Kubernetes.

We will develop an API containing single endpoint.

We will dockerize that API. For this, we will create a dockerfile.

We will send the image we have created with the dockerfile to GCR(Google Cloud Container Registry).

We will start three containers from this image.

We will create "Load Balancer" service, which will distribute the requests coming to the application to the containers.

I want to mention five concepts in Kubernetes.

Cluster

Cluster is a kubernetes server. In the process of creating Cluster, same introductions are applied like how many "master" and " node" will be.

Pod

Pod represents the application. It means that the application runs on pods. It corresponds to the container in Docker.

For more information, you can visit the address below.

https://cloud.google.com/kubernetes-engine/docs/concepts/pod

Deployment

Deployment allows pods to be managed. We define by using the deployment how many pods to be started, which images the pods will use, which environment variables to be assigned with these pods.

For more information, you can visit the address below.

https://cloud.google.com/kubernetes-engine/docs/concepts/deployment

Service

Service allows to make the applications running with pods accessible from outside. We will configure our service as "Load Balancer".

For more information, you can visit the address below.

https://cloud.google.com/kubernetes-engine/docs/concepts/service

Namespace

It provides isolation in the cluster. It allows pods, deployments and services to come together under a "namespace". It allows the applications to be isolated from each other in case of more than one application is in the cluster. It means that each application runs under a namespace which belongs itself.

For more information, you can visit the address below.

https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

2. Preparing the Development Environment

In this part, we will arrange the development environment.

Requirements

Visual Studio Code
Docker Desktop
Google Cloud Account
Google Cloud SDK

2.1. Installing the Docker Desktop

The Docker Desktop is needed to be installed in the machine. You can download the application from the address below and install the application just by clicking next. :)

https://www.docker.com/products/docker-desktop

2.2. Opening a Google Cloud Account

You need to open a Google Cloud account with your Google account. When you open a cloud account, a promotion is defined to allow us to use the application within a year. You can do tryings by using the promotion.

You can open a Google Cloud account from the link below. After opening the account, a band will appear above the page. You can activate the promotion from the activate button.

https://console.cloud.google.com/?hl=tr

2.3. Installing the Google Cloud SDK

Since you will run the commands on your local machine, the Google Cloud SDK needs to be installed. You can download the application from the address below and install the application just by clicking next. :)

Note: In the fifth step, when you are asked for the components to be installed, you need to activate "The Beta Commands" option.

https://cloud.google.com/sdk/docs/quickstart-windows

The "gcloud init" commad will run automatically when the Google Cloud SDK has been installed. Authorization will be made for you to access Google Cloud from your local machine. For this reason, you will need to enter your google account information.

You can say no when you are asked if you want to create a project. We will create the project ourselves.

2.4. Introducing the GCR

In order for the images to be sent to the GCR instead of the DockerHub, we need to make configuration. You must run the following command from the terminal.

gcloud auth configure-docker

3. Preparing the GCP

3.1. Activating the Kubernetes API

Activating the Kubernetes Engine API, which is one of the Google Cloud APIs, is required. You must run the following command from the terminal.

gcloud services enable container.googleapis.com

3.2. Creating the Project

You need to create a project to be able to use Google Cloud services. You need to run the following command from the terminal. You can update PROJECT_ID and PROJECT_NAME fields by yourself.

gcloud projects create [PROJECT_ID] --name=[PROJECT_NAME]
gcloud projects create my-k8s-api-project-1 --name="My K8S API Project"

You can run the command below to see the project created.

gcloud projects list

3.3. Choosing the Project

You need to define the project you have created on google cloud config in your local machine. You need to run the command below from the terminal. Remember to update the PROJECT_ID field according to your information.

gcloud config set project [PROJECT_ID]
gcloud config set project my-k8s-api-project-1

You can run the command below to see whether the project has been added to the configuration.

gcloud config get-value core/project

3.4. Activating the Billing

Before creating a cluster, you must link a billing account to the project you have created. To learn your Account Id, you need to run the following command from the terminal.

gcloud beta billing accounts list

You need to run the following command to install the alpha commands of the Google SDK.

gcloud components install alpha

When the prcoess is complete, you need to run the following command to link the Account Id to your project. Remember to update the PROJECT_ID and ACCOUNT_ID fields according to your information.

gcloud alpha billing projects link [PROJECT_ID] --billing-account [ACCOUNT_ID]

You can run the command below from the terminal to see that the invoice has been successfully linked to your project. Remember to update the ACCOUNT_ID field according to your information.

gcloud beta billing projects list --billing-account=[ACCOUNT_ID]

3.5. Creating the Cluster

You need to run the command below to create cluster. Remember to update CLUSTER_ID and PROJECT_ID fields according to your information.

gcloud beta container clusters create [CLUSTER_ID] --project=[PROJECT_ID] --addons=Istio --istio-config=auth=MTLS_STRICT --cluster-version=latest --machine-type=n1-standard-2 --num-nodes=1 --region europe-west1

gcloud beta container clusters create my-k8s-api-cluster-1 --project=my-k8s-api-project-1 --addons=Istio --istio-config=auth=MTLS_STRICT --cluster-version=latest --machine-type=n1-standard-2 --num-nodes=1 --region europe-west1

You can run the comand below to see the cluster created.

gcloud container clusters list

All preparations have been done. Now you can publish the project in the Kubernetes.

4. Publishing the application

I have created a simple API. You can clone this API from Github with the command below.

git clone https://github.com/ahmetkucukoglu/aspnetcore-k8s-sample.git
cd aspnetcore-k8s-sample

4.1. Creating the Dockerfile and Sending the Image to GCR

Create a .Dockerfile file in the project.

You need to create an image in the docker with the following command. You can determine the IMAGE_ID field by yourself.

docker build -t [IMAGE_ID] -f K8SSampleAPI/Dockerfile .
docker build -t my-k8s-api-image:1.0 -f K8SSampleAPI/Dockerfile .

Create a tag firstly to send this image we have created to GCR. Don't forget to update the LOCAL_IMAGE_ID, PROJECT_ID and IMAGE_ID fields according to your own information.

docker tag [LOCAL_IMAGE_ID] gcr.io/[PROJECT_ID]/[IMAGE_ID]
docker tag my-k8s-api-image:1.0 gcr.io/my-k8s-api-project-1/my-k8s-api-image:1.0

Send the image to GCR using this tag.

docker push gcr.io/[PROJECT_ID]/[IMAGE_ID]
docker push gcr.io/my-k8s-api-project-1/my-k8s-api-image:1.0

You can run the command below from the terminal to see the image sent to GCR.

gcloud container images list

4.2. Creating the Namespace

I mentioned the namespace in the first part. Create a namespace.yaml file in the project.

On the 4th line, define a name for the namespaces.

On the 6th line, define the label. We use the labels as tags. You can define a label as a key/value as you want.

Now, define this namespace in the Kubernetes cluster.

kubectl create -f K8SSampleAPI/namespace.yaml

You can run the command below from the terminal to see the namespace created.

kubectl get namespaces --show-labels

4.3. Creating the Deployment

In the first part, I mentioned deployment. Create a deployment.yaml file in the project.

On the 7th line, determine how many pods will be removed.

On the 11th line, define the label again. Determine the name of the application with the app key. You will need that while defining service to the defined value.

Give the image you have sent to the GCR to the image field on the 15th line.

Now, define the deployment in the Kubernetes cluster.

kubectl create -f K8SSampleAPI/deployment.yaml

You can run the command below from the terminal to see the deployment created.

kubectl get deployments --show-labels --namespace development

When deployment is done, three pods will run. You can run the command below to see.

kubectl get pods --show-labels --namespace development

4.4. Creating the Service

In the first part, I mentioned the service. Create a service.yaml file in the project.

On the 9th line, write the value in the app key that we defined when defining deployment. In this way, the requests coming to service are directed to the pods in the defined deployment. We have mentioned above that three pods will be started in deployment.

On the 12t line, define that the application will be published from the port 1453.

Now, define the service in the Kubernetes cluster.

kubectl create -f K8SSampleAPI/service.yaml

You can run the command below from the terminal to see the created service.

kubectl get services --show-labels --namespace development

When you run the command, you can see that the External-IP section is in the pending state. It may take up to a minute to assignment IP for the application. When the IP assignment is completed, it means that the application is ready to be published on the port 1453.

K8SSampleAPI

We have specified ApiKey and ApiSecret information in the deployment.yaml. We defined the values as environment variable. We have seen that these variables are successfully accessed within the application.

I determined the application Id information in the Startup file while the application was starting. Since three applications are started in Deployment, each application has Application Id information according to itself. If you make a request from different browsers, you will see that these three different ApplicationId are returning. In this way, we have seen that the requests are directed to three different pods.

You can run the following command to delete the cluster. When the cluster is deleted, namespace, deployment, service and pods will be deleted. Remember to update the CLUSTER_ID field according to your information.

gcloud container clusters delete [CLUSTER_ID] --region europe-west1
gcloud container clusters delete my-k8s-api-cluster-1 --region europe-west

You can access the last version of the project from Github.

ahmetkucukoglu / aspnetcore-k8s-sample

k8s-sample

View on GitHub

Good luck.

Developing AWS Serverless RESTful API

Ahmet Küçükoğlu — Mon, 27 Jan 2020 06:51:02 +0000

This article was originally published at: https://www.ahmetkucukoglu.com/en/developing-aws-serverless-restful-api/

1. Introduction

This article series is about how to develop RestfulAPI with a serverless approach. We will use AWS as a cloud provider.

In this part, i will show the scenario and give information about what will be the result.

Our RESTful API endpoints will be like below.

[POST] api/ads

[PUT] api/ads/{id}

[DELETE] api/ads/{id}

[GET] api/ads

[GET] api/ads/{id}

We will use the following AWS services.

API Gateway : It will provide the endpoints we need for the RESTful API.

Lambda : It will provide to write function for the GET, POST, PUT and DELETE requests.

DynamoDB : It will provide NoSQL solution. In the POST, PUT and DELETE requests, we will write the data to the DynamoDB.

ElasticCache : It will provide the distributed cache solution. In the GET request, we will read the data from Redis.

S3 : It will provide our code to be versioned and stored after each deployment.

CloudFormation : It will provide us to create and manage automatically all services mentioned above.

When the project is done, the architecture on AWS will be like below.

As a result, the record will be sent to the DynamoDB which is NoSQL solution of AWS in the POST request. Record changes in our DynamoDB table will be sended on the ElasticCache which is the solution service of the distributed cache of AWS. The data in the GET request will be read from ElasticCache instead of the DynamoDB.

2. Preparing the Development Environment

In this part, we will arrange our development environment.

Requirements

Visual Studio Code
AWS Account
AWS CLI
Serverless Application Framework

2.1. Creating a User in AWS Console

Creating an authorized user in AWS is required to make deployment with serverless.

Go to IAM from AWS Console. Choose "User" on the left menu. Click the "Add user" button.

Write "serverless" in the username fied. Choose "Programmatic Access" from the Access type section. Go to the second step(Permissions).

Choose "Attach existing policies directly" from the Permissions section. Give the following policies. Go to the third step(Tags).

AWSLambdaFullAccess
IAMFullAccess
AmazonS3FullAccess
CloudWatchLogsFullAccess
AmazonAPIGatewayAdministrator
AWSCloudFormationFullAccess
AmazonDynamoDBFullAccess
AmazonElastiCacheFullAccess

Write "Application" in the Tag Key field and "Serverless RESTful API" in the Tag Value field. Go to the fourth step(Review).

Click the "Create User" button.

On the screen, you will see the information about Access Key Id and Secret Access Key. Keep them in a side.

2.2. The Installation of AWS CLI

Open Command Prompt and run command line below.

aws configure --profile serverlessuser

AWS Access Key ID : The Access Key Id you will copy

AWS Secret Access Key : The Secret Access Key you will copy

Default region name : eu-central-1

Default output format : json

2.3. The Installation of the Serverless Framework

We will use the Serverless Application Framework to develop this application. You can visit the address below for the installation.

Serverless Framework - AWS Lambda Guide - Installing The Serverless Framework

3. Preparing the Endpoints

In this part, we will create the project and make deployment in a simplest way. We will not send a record to the DynamoDB yet. We will GET response after sending request to the endpoints.

Create the project folder with the command line below.

mkdir serverless-api && cd serverless-api

Create the serverless project with the command line below.

serverless create --template aws-nodejs --name ads-api

Open the project with the command line below via Visual Studio Code.

code .

Update the serverless.yml file like below.

serverless.yml v1

Delete the handler.js file. Create 4 files to be named as "create.js", "update.js", " delete.js" and "getByld.js".

create.js v1

update.js v1

delete.js v1

getAll.js v1

getById.js v1

Create a file named "create_request.json" in the project folder and paste the json in it. Defines the POST request model. Sets required fields.

create_request.json

Create a file named "update_request.json" in the project folder and paste json in it. Defines the PUT request model. Sets required fields.

update_request.json

Make deployment to AWS by using the commad line below.

serverless deploy

After the deployment, in a simplest way, lambda functions and the API endpoints to make request functions is created.

After the deployment process, will write 5 API endpoints and 1 API Key on the console.

You can see the result by sending request to the endpoints. You need to add "x-api-key" to the Headers and for the value, write API Key showed in the console.

Let's look at what changed in AWS after the process.

Stack is created in the Cloud Formation.

Our code is sent to S3.

API Gateway is created for the endpoints.

Lambda functions are created.

4. DynamoDB Integration

In this part, we will write the data coming over the API to DynamoDB. We will read the data from DynamoDB.

Install the required packages by using the command lines below.

npm i aws-sdk
npm i uuid

Update the serverless.yml file like below.

serverless.yml v2

Update the files of create.js, update. js, delete.js, getAll.js and getByld.js like below.

create.js v2

update.js v2

delete.js v2

getAll.js v2

getById.js v2

Make the second deployment to AWS with the command line below.

serverless deploy

You can see the result by sending request to the endpoints. You need to add "x-api-key" to the Headers and for the value, write API Key showed in the console.

Let's look at what changed in AWS after the process.

A table is created in the DynamoDB.

5. Redis Integration

In this last chapter, we will transfer the streams in the DynamoDB(adding, udating and deleting) to ElastiCache Redis and do reading from Redis in the GetAll request.

Install the required packages with the command lines below.

npm i redis

Before starting to make update in the files, you need to learn the subnetlds by entering AWS console.

Update the serverless.yml file like below. You need to update the subnetlds parts with your own information.

serverless.yml v3

Update the getAll.js file like below.

getAll.js v3

Create a new file named trigger.js.

trigger.js

Make the third and last deployment to AWS with the command line below.

serverless deploy

You can see the result by sending request to the endpoints. You need to add "x-api-key" to the Headers and for the value, write API Key showed in the console.

Let's look at what changed in AWS after the process.

ElasticCache is created.

Trigger is assigned to the DynamoDB.

You can access the final version of the project from Github.

ahmetkucukoglu / serverless-restful

Serverless RESTful on AWS

serverless-restful

Serverless RESTful on AWS

View on GitHub

Good luck.