DEV Community: Mạnh Đạt

I Built An Animation Tool With Angular!

Mạnh Đạt — Fri, 09 Sep 2022 09:18:44 +0000

Motivation to create

Recently, I blogged more than I used to. Most of the posts are tutorials.

I tried to add images, diagrams when possible. However, I think videos work best when describing procedures.

Diagrams are cool but it would be even more awesome if objects on diagrams can move :D.

So I created a tool to do just that: Helps me create videos to explain processes quickly.

You may say: Hey, there are many great video tool out there that you can learn in a few hours and they can do more than what you need!

Yes, I know. Great tools (even free) are out there but being able to create something outweigh all reasons.

I ended up spending mostly a week creating the first "usable" version.

The app

So here it is:

https://vanim.datmt.com/

I created a video showing how to use the app here:
https://www.youtube.com/watch?v=RE6RbT69VS0

You can go there and start playing with the tool. You can create some shapes, use images to create moving diagrams...

Here is one example I created to explain the concept of CountDownLatch in Java

Future of the app

I have some ideas in mind. The main purpose is to make creating videos even easier. I know the app is primitive now but I can now create some nice animation for my blog :).

Your suggestions are highly appreciated!

Special thanks

This app is not possible without the following projects:

Fabricjs http://fabricjs.com/: It is the engine that run the app. Thanks!
Jenkins: Thanks to this awesome projects, I can literally push new changes online in just a single click.

Thanks for your attention. Have a great weekend and hope you have fun creating new things!

Java Concurrency: Threads communication tutorial

Mạnh Đạt — Wed, 07 Sep 2022 05:08:42 +0000

Overview

Java provides mechanisms for threads to communicate with each other when working on common resources. This post gives you a concrete example of threads communication.

Threads communication example

Let's consider this scenario. Alice is a computer programmer who also has a love for buying new shiny things (phones, laptops, gadgets). She has a list of things to buy. She works at a company called MonkeyTypes Inc. Her paycheck is $1,000 monthly.
Imagine she currently has this wishlist:

A new Macbook: $3000
A new mechanical keyboard: $400,
A new phone: $500,
A new shiny gadget: $500

She wants to have all of them in no particular order.
Her current balance is $0. What she wants is when she gets her monthly salary, she would spend the money in her balance to buy any of those things.
Let's see how we can convert this scenario to demonstrate thread communication.

Code implementation

For this scenario, let's create two Runnable tasks: One to simulate Alice paycheck and other to simulate her purchase.
We also need a class to represent her bank account.
Let's create the bank account class first:

class BankAccount {
    private int balance = 0;


    private static Lock lock = new ReentrantLock();
    private static Condition paycheckArrivedCondition = lock.newCondition();

    public void getPaid(int amount) {
        lock.lock();
        try {
            System.out.println("Getting paid " + amount);
            balance += amount;
            paycheckArrivedCondition.signalAll();
        } finally {
            lock.unlock();
        }
    }


    public void withdraw(int amount, String purpose) {
        lock.lock();
        try {
            while (balance < amount) {
                paycheckArrivedCondition.await();
            }
            System.out.println("Withdraw " + amount + " to " + purpose);
            balance -= amount;

            System.out.println("new balance -> " + balance);
        } catch (InterruptedException ex) {
            ex.printStackTrace();
        } finally {
            lock.unlock();
        }
    }
}

As you can see, this class has one field: balance to hold the current balance. Also, there are two methods to deposit and withdraw money to and from the balance.
The most interesting details here are the lock and the condition. I created a static lock and a static condition at the beginning of the class. The lock, as you may know, helps synchronize access to the balance. The condition, on the other hand, helps make communication between threads possible.

The withdraw() method

At the beginning of the withdraw method, the lock method on the ReentrantLock instance is called. This ensures that only the thread that has the lock can execute the code in this function.
Next, the try/catch/finally blocks make sure the lock is released at the end.
The while loop checks if the balance has enough money, if not, the await function is called on the condition instance.** This call releases the lock.**

The deposit() method

Similar to the withdraw() method, threads need to acquire the lock to execute code here. One interesting thing about this method is the call to the method signalAllon the condition instance. This call is the meat of thread communication. This wakes up all the waiting threads and the check for balance > amount starts again.

The Runnable class to deposit money

Now the BankAccount class is available, let's create a runnable class to deposit money to a BankAccount instance:

class PayEmployee implements Runnable {

    private final BankAccount bankAccount;
    private final int amount;

    PayEmployee(BankAccount employeeBankAccount, int amount) {
        this.bankAccount = employeeBankAccount;
        this.amount = amount;
    }

    @Override
    public void run() {
        bankAccount.getPaid(amount);
    }
}

The Runnable class to withdraw money

class BuyThings implements Runnable {
    private final BankAccount bankAccount;
    private final String purpose;

    private final int amount;

    public BuyThings(BankAccount account, String purpose, int amount) {
        this.bankAccount = account;
        this.purpose = purpose;
        this.amount = amount;
        System.out.println("Plan to " + purpose + " with " + amount);
    }

    @Override
    public void run() {
        bankAccount.withdraw(amount, purpose);
    }
}

Alice buys things in action

Now let's implement the code where Alice submits her wishlist.

public static void main(String[] args) {
        BankAccount myAccount = new BankAccount();
        var executors = Executors.newFixedThreadPool(5);
        executors.submit(new BuyThings(myAccount, "buy new macbook pro", 3_000));
        executors.submit(new BuyThings(myAccount, "buy new phone", 500));
        executors.submit(new BuyThings(myAccount, "buy new keyboard", 400));
        executors.submit(new BuyThings(myAccount, "buy new gadgets", 500));

        int cycle = 6;
        while (cycle > 0) {
            try {
                Thread.sleep(5000);
            } catch (InterruptedException ex) {
               ex.printStackTrace();
            }

            executors.submit(new PayEmployee(myAccount, 1_000));
            cycle--;
        }

        executors.shutdown();
    }

From line 4 to line 7, all her purchases are submitted.
From line 9 to line 19, I simulate her payment. Let's say her contract with the company has only 6 months left. Let's run the program and see the output:

As you can see, after getting paid the first $1000, Alice buys a phone and then a new keyboard ... However, this order is not consistent. The next run may produce different order. One certain thing is a MacBook always gets purchased last because only after the next-to-final payment, Alice have enough money to afford this.

You may ask, what if Alice only has 4 payment cycles left instead of 6? That means she never has enough money for a MacBook. In such a case, the program will run forever because the buy Macbook thread keeps waiting for the condition to meet. (so sad :( )

Conclusion

In this post, I have introduced you to the concept of thread communication in Java using Lock and condition. A thread can acquire the lock, and check if the condition is met. If the condition is not met, a call to await on the condition instance releases the lock for other threads. A thread can notify all other threads by using signalAll (or signal to notify a random thread).
The code for this post is available here on Github

This post is a part of the Java concurrency tutorial series

MongoDB Aggregation Framework Cheat Sheet

Mạnh Đạt — Tue, 02 Aug 2022 10:14:18 +0000

Importing sample data

This post is a quick introduction to the MongoDB Aggregation Framework. It's a part of a mini-series called Mongo DB Cheatsheets
To follow the tutorial, you need to import the following data:

[
  {
    "first_name": "Larry",
    "last_name": "Freeman",
    "networth": 340,
    "investments": [
      {
        "name": "Oil Industry",
        "year": 1990,
        "value": 10
      },
      {
        "name": "Bank Industry",
        "year": 1995,
        "value": 5
      },
      {
        "name": "Bank Industry",
        "year": 1997,
        "value": 35
      },
      {
        "name": "Publishing Industry",
        "year": 1995,
        "value": 130
      },
      {
        "name": "Beer Industry",
        "year": 2005,
        "value": 40
      },
      {
        "name": "Energy Industry",
        "year": 2005,
        "value": 55
      }
    ]
  },
  {
    "first_name": "Bill",
    "last_name": "Freeman",
    "networth": 80,
    "investments": [
      {
        "name": "Oil Industry",
        "year": 1992,
        "value": 30
      },
      {
        "name": "Oil Industry",
        "year": 1995,
        "value": 10
      },
      {
        "name": "Bank Industry",
        "year": 1995,
        "value": 15
      },
      {
        "name": "Bank Industry",
        "year": 2005,
        "value": 25
      }
    ]
  },
  {
    "first_name": "Jerry",
    "last_name": "Whackman",
    "networth": 190,
    "investments": [
      {
        "name": "Publishing Industry",
        "year": 1990,
        "value": 130
      },
      {
        "name": "Beer Industry",
        "year": 1990,
        "value": 40
      },
      {
        "name": "Energy Industry",
        "year": 2005,
        "value": 15
      }
    ]
  },
  {
    "first_name": "Mary",
    "last_name": "Whackman",
    "networth": 280,
    "investments": [
      {
        "name": "Oil Industry",
        "year": 1997,
        "value": 30
      },
      {
        "name": "Bank Industry",
        "year": 2005,
        "value": 25
      },
      {
        "name": "Auto Industry",
        "year": 2008,
        "value": 125
      }
    ]
  },
  {
    "first_name": "Kate",
    "last_name": "Whackman",
    "networth": 120,
    "investments": [
      {
        "name": "Logistics Industry",
        "year": 2017,
        "value": 30
      },
      {
        "name": "Bank Industry",
        "year": 2005,
        "value": 50
      },
      {
        "name": "Auto Industry",
        "year": 2008,
        "value": 10
      }
    ]
  }
]

This sample is a list of 5 rich people from two families. The data contains their net worth and their investments over the years.
To import the data, open the MongoDB shell, select the database (starter) and do the import like this:

Now you are ready for some aggregations!

MongoDB Aggregation Overview

You can think of aggregation as a pipeline where your data goes through a number of steps. The output of one step is the input of the next step.

MongoDB Aggregation Quick example

From the list of rich people above, say, you want to get the people from the Whackman family, order the results by their net worth, and only display their name and net worth, ignoring other fields.
Then you can issue this aggregation query to achieve the result:

db.rich.aggregate([
  { $match: { last_name: "Whackman" } },
  { $sort: { networth: 1 } },
  { $project: { _id: 0, first_name: 1, last_name: 1, networth: 1 } }
])

As you can see, this aggregation has three stages:

$match: filter only matched documents. In the example, I filtered rich people who have a source as self-made
$sort: sort by field. (1 for ascending, -1 for descending)
$project: select fields to display (1 for showing, 0 for hiding)

The positions of the stages in the aggregation array determine the running order. That means in the example above, $match runs first, then $set and finally $project.

Using $match for Filtering

You can use $match as you can with find. This stage allows you to filter data to a smaller set.
In the list above, you can use $match to filter people from one family only. For example, let's get only people from the Freeman family:

db.rich.aggregate([
  { $match: { last_name: "Freeman" } }
])

You can also use $match to do more complex filters like so:
Filter documents that made investments in years later than 2006. Notice that this filter still outputs the full documents with investments before 2006 (if any). The only requirement is that document has at least one investment after 2006.

db.rich.aggregate([ 
   { $match: { "investments.year" : { $gt: 2006} } }
])

Using $project

You use $project to modify fields, create new fields, and show and hide fields.
Here are some examples:

Display people with their first name, last name and net worth only:

db.rich.aggregate([
  { $project: { _id: 0, first_name: 1, last_name: 1, networth: 1 } }
])

In this example, for fields I want to show, I set the value to 1. I need to hide _id field explicitly. Other fields are hidden if there are no settings for them.

Adding a new calculated field to the document

Let's say there is a 10% tax applied to these rich people's net worth and you want to display that field in the document. $project would handle that easily:

db.rich.aggregate([ 
  { $project: { _id: 0, first_name: 1, last_name: 1, networth: 1, calculated_tax: {$divide: ["$networth", 10]} } }
])

Using $unwind

I cannot put a better definition than the official of $unwind so here it is:

Deconstructs an array field from the input documents to output a document for each element.
https://www.mongodb.com/docs/manual/reference/operator/aggregation/unwind/

For example, you have this document:

{
    "name": "Jane",
    "hobbies": ["Reading", "Swimming"]
}

After running $unwind on this object, you will have two objects

[{
    "name": "Jane",
    "hobbies": "Reading"
},
{
    "name": "Jane",
    "hobbies": "Swimming"
}]

Let's say the requirement is to show only investments in the Publishing Industry made by all of the people.
From the requirements, I'm going to write an aggregation pipeline with two stages: $unwind and $match:

db.rich.aggregate([
  { $unwind: "$investments" },
  { $match: { "investments.name" : "Publishing Industry" } }
  ])

In this example, in the first steps, I used $unwind to deconstruct the investments array. As a result, for each investment, I have one document. If I only run the $unwind stage, here is the output:

As you can see, for one person (Larry Freeman for example), instead of one document, now he has multiple documents, each containing one of his investments.
The $match stage filters out all the investments from industries that aren't "Publishing Industry". Thus, I got the result as expected.

Using $group

$group is one of the most powerful operators in the aggregation framework. As the name suggests, you use $group when you want to group documents that have common traits into one document.

Here are some important things you want to know about $group:

You need to define a "group key", which is a common trait. This "group key" can be a field, a group of fields, or an expression
You define the group key using _id

Let's try an example with $group. Let's say I want to get the unique family names in the collections. From the beginning of this post, I already know that there are only two families: Freeman and Whackman

db.rich.aggregate([
  { $group: { _id: "$last_name" } }
])

Using accumulators in $group

Using accumulators helps you get some nice results with minimal code.
For example, you want to calculate the total net worth of each family. This is what you need to execute:

db.rich.aggregate([
  { $group: {_id: "$last_name" , total_networth: { $sum: "$networth"}}}
])

You can also use another function, $avg to calculate the average net worth of each family:

db.rich.aggregate([
  { $group: {_id: "$last_name" , total_networth: { $avg: "$networth"}}}
])

Using $sample to extract a portion of a large dataset

There are times you have to work with large datasets. That means the document count is in the millions range. In such cases, you may want to work with a portion of that collection, says 1,000.
The $sample operator helps you do that.
For example, the dataset in this post has 5 documents and I only want to work with 2, this is what I do:

db.rich.aggregate([
  { $sample: { size: 2 } }
])

Practical MongoDB aggregation

In this section, we are going to solve some questions using aggregations.

List the unique name of the industries

For this question, we are going to use $unwind and $group

db.rich.aggregate([
  { $unwind: "$investments"},
  { $group: {_id: "$investments.name"}}
])

Which family invests most in the Oil industry?

For this question, we will do these steps:

Use $unwind to deconstruct the investments
Use $match to filter out other industries.
Use $group to group the family name and $sum to calculate the total amount

db.rich.aggregate([
  { $unwind: "$investments" },
  { $match: { "investments.name" : "Oil Industry"} },
  { $group: { _id: "$last_name", total_investment: {$sum: "$investments.value" } } }
  ])

Rank people by the total value of their investments

In this task, we need to calculate the sum of each person's investments and then rank the value descendingly.
There are many ways to do this but this is one:

First, $unwind the investments array
Then $group by first name and last name, and use $sum to calculate the total investments' value
Use $sort to order the total value descendingly

db.rich.aggregate([ 
    {$unwind: "$investments"}, 
    { $group: { _id: ["$first_name", "$last_name"], total_investment_value: { $sum: "$investments.value" } } },
    { $sort : {"total_investment_value": -1} }
])

Calculate the total net worth, and average net worth of each family and rank them by average net worth

At this point, you probably know what to do.

Use $group and $sum, $avg to calculate the requested values
Use $sort to rank the average net worth value

db.rich.aggregate([
  { $group: { _id: "$last_name", total: {$sum: "$networth"}, average: {$avg: "$networth"}}},
  { $sort: {"average": -1}}
  ])

Rank the top 3 investments by value.

This is yet another task we need to use $unwind. Here are the steps:

Use $unwind to deconstruct the investment array
Use $sort to sort the investments by values
Use $limit to get only the top 3 items

db.rich.aggregate([
  { $unwind: "$investments" },
  { $sort: { "investments.value": -1 } },
  { $limit: 3}
  ])

Which year has the most investment amount?

To solve this question, we need to use $unwind, $group, $sort, and $limit

db.rich.aggregate([
  { $unwind: "$investments" },
  { $group: { _id: "$investments.year", amount: {$sum: "$investments.value"}} },
  { $sort: {"amount": -1} },
  { $limit: 1}
  ])

Count the number of people who have a total investment value greater than 50 in each family

To complete this request, we need to do quite many things:

Use $unwind to deconstruct the investment array
Use $group by _id and $sum to calculate the total investment value of each person. In this step, we also use the accumulator $first to extract the family name and store it in a field called family_name
Use $match to filter out people with less than 50 investment values
Use $group on last name and $sum to count the number of the remaining people.

db.rich.aggregate([
  { $unwind: "$investments" },
  { $group: { _id: "$_id", family_name: {$first: "$last_name"}, "invest_value": { $sum: "$investments.value"} } },
  { $match: { "invest_value" : { $gt: 50} } },
  { $group: {_id: "$family_name", "people_count": {$sum: 1}}}
])

Conclusion

In this post, we have learned some of the most common usages of the MongoDB aggregation framework. This post is by no mean a complete reference. You need to refer to the official documentation for a complete list.

MongoDB Basic CRUD Cheat Sheet

Mạnh Đạt — Thu, 28 Jul 2022 18:32:22 +0000

If you have been doing software development for some time, you'd probably know what CRUD is. CRUD stands for CREATE-READ-UPDATE-DELETE. Those are the day to day task we do on databases, regardless of their types.
Let's get started by using a new database. As you may remember from the previous post, there is no "create" command in MongoDB. We simply use the use command. I'm going to call my new database starter

use starter

Collection CRUD

Create Collections

Before inserting documents, you need to create collections. You can create a collection by:
null
Let me demonstrate

db.createCollection("create_explicitly")
db.create_implicitly.insert({})

On line 1, I called the createCollection explicitly and gave that function a name.
On line 2, I assumed the collection named create_implicitly exist and inserted an empty document.
In both cases, the collections were created successfully

Rename collection

To rename a collection, simply call renameCollection on the collection you want to rename

# Create a wrong name collection
db.createCollection("wrong_name")

# Rename the collection
db.wrong_name.renameCollection("correct_name")

Drop collection

To remove a collection (and all documents inside), simply call:

db.create_implicitly.drop()
db.create_explicitly.drop()

Document CRUD

In this section, we are going to learn how to do basic CRUD on documents

Create document

You have three functions to create documents in MongoDB. They are insert, insertOne and insertMany

Using insert

db.dogs.insert({"name": "Bingo"})
db.dogs.insert([{"name": "Pie"}, {"name" : "Borg"}])

As you can see, you can insert one document or many documents using insert.

Using insertOne

As the name suggest, insertOne inserts only one document. Thus, it accepts only objects, not array

db.dogs.insertOne({"name": "Biscuit"})

Using insertMany

Similar to insert, you can insert multiple documents using insertMany. This function accepts only array, not object:

db.dogs.insertMany([{"name": "Steven"}, {"name": "Jessica"}])

You may wonder, what are the differences between insert and insertOne and insertMany?
Take a look at the screenshots, return types are different.

Read Documents

Now we have some documents, let's try getting them

Using find

The most common way to retrieve documents in MongoDB is using find

db.dogs.find()

You can pass a filter object to the find function to get exactly what you need.
For example, finding the dog with name "Bingo" looks like this:

db.dogs.find({"name" : "Bingo"})

You are not limited to exact matching. For example, you can use regex to search for documents.
Let's find all the dogs with names begin with "B"

db.dogs.find({"name" : {$regex: /B.*/i}})

There are many other powerful queries you can do with find. However, that's a topic for another post. We only do basic CRUD here.

Update documents

In the update command, you need to specify the query to get the matched documents and the operation you want to do on those documents.

Update one document

You can update a document by using its ObjectId.

db.dogs.update(
  {_id:  ObjectId("62e2533531ed3d24b2e3e0df") }, 
  { $set: { age: 5 } }
)

In this example, I knew that the dog named "Bingo" has that specific ObjectId. So, I passed that in the query object and operate the $set operation on its age field. I queried the dog "Bingo" again, I could see the age field is set to 5.

Update a document by ID
In case your query returns multiple documents and you only want to update the first one, use updateOne or update

db.dogs.updateOne(
  {name:  {$regex: /B.*/i } }, 
  { $set: { age: 8 } }
)

Update multiple documents

You can use the update command to execute update on multiple documents, not just one

db.dogs.update(
  {name:  {$regex: /B.*/i } }, 
  { $set: { age: 6 } },
  {multi: true}
)

As you can see, I needed to set {multi: true} to update multiple documents. By default, the update function only updates one record.
If I try to find all the dogs with name starts with B, I can see they all have age set to 6:

You can also use updateMany to update multiple documents without passing {mulitple: true}

db.dogs.updateMany(
  {name:  {$regex: /B.*/i } }, 
  { $set: { age: 7 } }
)

Delete documents

Similar to update, you also need to write a query to find the matched documents and call delete on them.

Delete one document

Use deleteOne to delete one document from the list of matches. That means, even if you have more than one document, there is only one will be deleted:

db.dogs.deleteOne({name: "Bingo"})

Delete all matched documents

If you want to delete all matches documents, use deleteMany

db.dogs.deleteMany({name: {$regex: /B.*/i}})

In the example above, I first find the dogs that have names start with B. As you can see, there were two result.
After issuing the deleteMany query, none was found.

Delete all documents

If you want to delete all documents in a collection, simply call deleteMany passing and empty filter object. This is similar to the command TRUNCATE in SQL databases.

Conclusion

In this post, I have listed some of the most common CRUD commands in MongoDB. The list is far from comprehensive but it is a good start for people who have just learned this NoSQL database.

Setup MongoDB Test Environment With Docker

Mạnh Đạt — Thu, 28 Jul 2022 18:30:47 +0000

This tutorial helps you quickly setup a MongoDB instance so you can start learning this topic right away.
We are going to use MongoDB 5 in this series. I haven't tested with other versions so if you are from the past or the future, make sure you use the correct version as in this post.

Running MongoDB with docker

If you have Docker installed, running MongoDB is really easy:



docker run -dit --restart=always -p 27017:27017 mongo:5.0

Here, I mapped the port 27017 of MongoDB to the port 27017 of the host. If your host doesn't have the port 27017 available, change that number to a different number.
You only need to do the port mapping if you want to connect to the instance using clients such as MongoDB Compass. If you use the shell (as I'm going to), you just need to run this:



docker run -dit --restart=always mongo:5.0

I run the container in detach mode and make it always restart for convenient.
Let's check if the instance is running:



docker ps

As you can see, I have already setup one instance and it's available there.
You should be able to see your instance as well.

Connect to the MongoDB Shell inside Docker container

As mentioned above, I'm not going to use a client but the MongoDB shell inside the docker container. From the docker ps command's output, I know that the container id is: bbc5....
So, I can execute inside the container by issuing the following command:



docker exec -it bbc5a637b0e6 bash

Inside the container, type mongo and you can access the mongo shell

MongoDB Hierachy

In MongoDB, at the top level, you have databases. Inside databases, there are collections. Inside collections, you have documents.
Documents are JSON objects that contain keys and values.

Basic MongoDB commands cheat sheet

To clear the screen type:

cls

And hit enter
To show available databases, type:



show dbs

To use a particular database, type:



use <dbname>

One important note here: There is no create database command in MongoDB. You can "create" one by simply using the use command.
To show all documents in a collection, type:



db.<collectionName>.find()
# For example:
db.friends.find()

Conclusion

In this post, I've shown you how to setup MongoDB and do some basic queries. We are going to learn more about MongoDB in the next articles.

Spring Cloud Config Server Complete Guide

Mạnh Đạt — Fri, 22 Jul 2022 12:13:31 +0000

Overview of Spring Cloud Config Server

Spring Cloud config server is a REST application that is built on top of Spring Boot. The main purpose of a config server is to store the configurations for all services in an application (think microservices). Thus, each service doesn't need to store its configurations. They just need to point to the config server to get their configurations.

Spring Cloud Config Server can work with various technologies to deliver the configurations in a secure way such as HashiCorp Vault, Git, or file system (less secure, not suitable for most production environments).
Let's get started with an example.

Spring Cloud Config Server Quick Start

Let's say we are building a microservice application for a restaurant chain. One of the services is called cook service. All it does is report the number of cooks available in the whole chain. For simplicity's sake, we are going to build a Spring Boot application with just one controller like this:

package com.datmt.springcloud.cookservice.controller;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/")
public class CookController {


    @Value("${cook.count}")
    int cookCount;

    @GetMapping
    public String cook() {

        return "Cook count: " + cookCount; 

    }

}

As you can see, this service gets the number of cooks in an application properties variable called cook.count. Normally, we would set such properties in the service application.properties file. However, today, we will get it from somewhere else: Spring Cloud config server!
There are many ways to define configurations for services in the config server (which we are going to discover later). In this section, we will read the configuration from the file system.
Let's take a look at configuration of our Spring cloud config server:
Let's pay attention to the last 2 lines of this config file. In line 3, we specified the active profile for configuration file scan, which is native. That means spring cloud configuration server will scan the file system. Line 4 specifies the exact location to search. The classpath refers to resources directory in our project:

In the cook-service*.properties files, there are one setting that has different values for different environment:

In the config folder, there are three files for the cook service. Why are these files named like this? By convention, configurations files for services have this format: servicename-env.yaml or servicename-env.properties.
That means the service name here must match the name in the cook service application.properties
The configuration file of the cook service:

server.port=9977
spring.application.name=cook-service
spring.profiles.active=dev
spring.cloud.config.uri=http://localhost:9988
spring.config.import=optional:configserver:http://localhost:9988

As you can see, on line 2, we named the service as cook-service. Line 3 specify the active profile, which is dev. That means the configurations for cook service will be read from the file config/cook-service-dev.properties on the config server. In case that file is not available, the content of the file config/cook-service.properties will be used.
Let's start the config server and see what we have:

As you can see, the configuration URL for dev and prod profiles includes the default configurations. For a nonexistent profile, the default config will be served.
Now, let's start the cook service to see if it can get the right configuration. Remember, the active profile is dev so we would expect to see the number 190:

If we change the active profile to prod

server.port=9977
spring.application.name=cook-service
spring.profiles.active=prod

and reload the cook service, the output will change:

As you can see, we have successfully gotten the configuration for the cook service from the config server.
In the next section, we will try to reload the configuration when there are updates on the config server without reloading the cook service.

Refreshing configuration with Actuator

If there are changes in the config server, how can you update the cook service so it will get the correct values without restarting?
Spring Actuator is the answer.
In the pom.xml file, add Spring Actuator as a dependency:

<dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>

      <!-- Omit other dependencies -->
    </dependencies>

In the service application.properties file, add the following line:

management.endpoints.web.exposure.include=refresh

In the controller, add the annotation @RefreshScrope to the class:

package com.datmt.springcloud.cookservice.controller;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/")
@RefreshScope
public class CookController {


    @Value("${cook.count}")
    int cookCount;

    @GetMapping
    public String cook() {

        return "Cook count: " + cookCount;  

    }

}

By adding these configurations, we can now call the cook service at /actuator/refresh to request a refresh.
For example, for the current running environment (prod), we change the value 4000 to 5000 and then send the following request:

As you can see, the service responds back with the config that changed.
If we reload the cook service now, the value should be updated.

Configure Spring Cloud Config Server With Git

Let's find out how you can use git to store your configurations and let the config server pull the data and serve it to other services.

Using Public Git

Let's say we have a repository on GitHub storing the config files like this:

The content of those files is exactly the same as the ones we have on our local machine. Let's configure the config server to use git.
The configuration is quite simple, you only need to make these changes to the application.properties file in the config server:

server.port=9988
spring.application.name=config-server
spring.profiles.active=native, git
spring.cloud.config.server.native.search-locations=classpath:/config
spring.cloud.config.server.git.uri=https://github.com/datmt/quick-samples.git
spring.cloud.config.server.git.search-paths=config-server/config

Pay attention to lines number 3, 5, and 6. On line 3, we added git as an active profile without removing the native option.
You may wonder, how does the config server choose which profile to use? The answer is the last one on the list.
On line 5, we point to the Git repository.
On line 6, we tell the config server the path to the config files.
Let's restart the config server and see if it can read the config:

As you can see, the config server was able to read the configurations from git without problems.
Let's change a value on git and see if the output is updated. Let's say we change the cook.count value from 5000 to 3000:

Just by reloading the browser, we can see the configuration has changed!

Isn't this awesome? Of course! But who would store sensitive data like configuration in a public git? Let's add some authentication, shall we?

Adding Git authentication for security

Let's switch the git repository to a private one. Now, the configuration looks like this:

server.port=9988
spring.application.name=config-server
spring.profiles.active=native, git
spring.cloud.config.server.native.search-locations=classpath:/config
spring.cloud.config.server.git.uri=https://github.com/datmt/Quick-Sample-private.git
spring.cloud.config.server.git.search-paths=config-server/config

If we launch the config service now and visit the configuration URL of the cook service, we would see an error page:

This is because the config server is unable to read the files from GitHub since the repo is private.
How can we configure it so the config server can read the data? We are going to use GitHub personal access token!
You can follow the guide here to create one: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token
After creating the token, we will put the details in the config server application.properties

server.port=9988
spring.application.name=config-server
spring.profiles.active=native, git
spring.cloud.config.server.native.search-locations=classpath:/config
spring.cloud.config.server.git.uri=https://github.com/datmt/Quick-Sample-private.git
spring.cloud.config.server.git.search-paths=config-server/config
spring.cloud.config.server.git.username=my_temp_access_token
spring.cloud.config.server.git.password=ghp_D6AlGb0f9CY......D5XG0UqzJB

As you can see on lines 7 and 8, both username and password are needed. The password is the access token, obviously. The username is the name of the token.
Now, if we reload the Spring cloud config server, we can see the settings again:

Enable Configuration Encryption for Security

Up until now, all the configurations are stored in plaintext. This is not the best practice.
Let's add some security layer by enabling encryption.
Spring cloud config server supports both symmetric and asymmetric encryption. We are going to implement symmetric encryption in this post.
Let's add encryption settings in the config server application.properties file:

server.port=9988
spring.application.name=config-server
spring.profiles.active=native, git
spring.cloud.config.server.native.search-locations=classpath:/config
spring.cloud.config.server.git.uri=https://github.com/datmt/quick-samples.git
spring.cloud.config.server.git.search-paths=config-server/config
encrypt.key=SUPER_SECRET_ENCRYPT_KEY

On line 7, we put the settings for encryption with the encryption key. In production settings, this is a bad practice. You should at least store the encryption key as an environment variable so it is not exposed in the source code.
With these new settings, we can send a POST request to the config server /encrypt to encrypt data. For example, if we want to encrypt the number 5000, the request should be like this:

curl -X POST -H "Content-Type: text/plain" http://localhost:9988/encrypt -d=5000

Here is the output:

Now let's put that value to our git:

Conclusion

In this post, we have learned how to use the Spring cloud config server to centralize configurations for all other services. It is a versatile solution that supports multiple mechanisms to store and manage data such as file system, git, HashiCorp Vault, JDBC

[2 days appwrite] Transfer text/Files between devices

Mạnh Đạt — Wed, 11 May 2022 20:07:56 +0000

Overview of My Submission

The app lets people share text/image/file between devices.

This grows from my need as I worked in a quite restricted environment (VPN, blocked sites, apps).

Every time I need to share some text between PC and mobile, I need to use Skype, which is very buggy.

So this app let me paste text on any device and get it on others.

It also works with files.

Here is the app URL:

https://x2.openexl.com/

Here is how the app work:

For example, you want to share a piece of text from your PC to your mobile, you paste that text in the Bury It section:

Then, you click on the button "Bury it". After a few seconds, you will get the code to use it on other devices:

On a difference device, you use that code and put into the Dig it section.

Hit the button and you'll see the text appear:

The same flow for transferring files.

Submission Category:

Web2 Wizards

Link to Code

There are two parts of the app. One web app (Angular) and api (written in Spring boot).

https://github.com/datmt/File-Bury-APP (Angular)
https://github.com/datmt/File-Bury-API (Spring boot)

Additional Resources / Info

This app is not possible without these projects:

https://github.com/appwrite/demo-todo-with-angular
appwrite
Spring boot

Fix Permission denied (public key) With Gitlab Windows

Mạnh Đạt — Fri, 06 May 2022 02:43:37 +0000

Recently I need to checkout a new repo from a self-hosted Gitlab server. I did the usual thing:

Generate a pair of ssh keys using ssh-keygen (since I'm on a new Windows machine). If you already have the ssh keys (try ls in your ~/.ssh and you see id_rsa and id_rsa.pub), then you don't need to generate the key pairs again.
Add it to the SSH key section of Gitlab

But still, I got the error:

Normally, it should work, right? Yes, I've done this countless time with GitHub, Gitlab, Bitbucket... and had no issue.

By the way, the account has 2FA enabled.

I don't know the reason for this but I have a workaround.

Access to Gitlab using personal access token

The workaround is to use Gitlab's personal access token. I haven't tried this on Github (since it works perfectly with SSH key) but I guess it should work too.

The first step is to generate a personal access token:

Then, enter the name of the token and check api

Then click on Create personal access token and you will get a token to use:

Now, head back to your repository and copy the standard https URL. It should look something like this:



https://git.some-server.com/group/repo/repo-name.git

You need to modify that URL a bit using your newly created token:



https://<token_name>:<token>@git.some-server.com/group/repo/repo-name.git

Now, I can use that url to clone the repo:

In case you already have the repo, you just need to update the remote url



git remote set-url origin https://<token_name>:<token>@git.some-server.com/group/repo/repo-name.git

Then you can pull, push normally.

Conclusion

That's it! You can continue working on your repo again. I still don't know why adding ssh didn't work but using the personal access token is an acceptable work around.

One thing I don't like about this method is any one who can access to your machine can run:



git remote -v

And see your access token. That person then can use the credentials to do what you can on your repo.

So, lock your screen when you are not at your computer :D

If you enjoy this post, make sure to check my blog out at https://datmt.com. I wrote about #programming stuffs there :)

Using Keycloak to authorize & authenticate Spring Boot application

Mạnh Đạt — Sun, 21 Nov 2021 10:38:54 +0000

Recently I started an SaaS project. For any project, one of the first problems you need to solve is authentication and authorization. It's 2021 and it's quite foolish to build your own system to log users in and check their role. After a few weeks study available options, I decided to go with Spring Boot & Keycloak.

If you don't know Keycloak, it's an opensource project that help you quickly setup authentication & authorization for your app. In my case, this is a perfect fit.

Project requirements

App has three roles member, moderator, admin
API must have endpoints to create user and let user login
Endpoints authorization could be configure with path prefix (for example /admin...) or by developer specifies the role allowed for each endpoint (using @RolesAllowed)

The code

The project is available here on Github, you can checkout and try it right away:
https://github.com/datmt/Keycloak-Spring-Boot-Login-Create-User

Need tutorial?

If you need step by step tutorial, check my post here:

https://datmt.com/backend/integrate-keycloak-with-spring-boot-step-by-step/

Conclusion

Keycloak is a mature solution for authorization and authentication. It has quite many customization that can fit into various needs. If you can use it in your project, it can save you a great deal of valuable time.

Create an Electron desktop app version of jwt.io

Mạnh Đạt — Thu, 01 Jul 2021 01:08:14 +0000

So last night, for some reasons I couldn't access jwt.io to decode a token (the website was loading very slowly) so I thought why didn't I create a desktop app so I don't have to suffer from this wait in the future?

Well, after a few hours of work, I got exactly (nearly) what I wanted:

Let me show you how I did it.

Step 1: Search for electron vue

I didn't work with electron much but I knew that it's based on Javascript so it's a good chance that somebody has already created a project that have electron bundle with Vue. My guess was right, I found this awsome repo on github:

https://github.com/SimulatedGREG/electron-vue

All credit to the author.
I cloned the repo and opened it in VSCode.

Step 2: Find a jwt decoder

Again, with google, I got this repo from Auth0 that help me decode jwt to get header and body:

https://github.com/auth0/jwt-decode
Again, thanks the developers.

So in my project, I ran

npm install jwt-decode

And I got the first working version.

Step3: Install codemirror for syntax highlighting

I used this repo:
https://www.npmjs.com/package/vue-codemirror

Thanks the developer :)

But only get basic function of a code editor functioning. I couldn't get syntax highlighting work yet (Have no idea why, hopefully you guys can give a pointer).

So, with the help of all the awesome developers of the repos I cloned, I got a working version of a jwt decode in just a few hours.

To be honest, the most time I spent was on trying different syntax highlighting libs and got no results :D

Conclusion

Building desktop app is fun and easy, especially for small projects like mine. You should give it a try.

Thanks for reading

Step by step to install kubernetes cluster

Mạnh Đạt — Mon, 07 Jun 2021 00:21:49 +0000

This post was originally posted here

Preface

I've been trying installing a Kubernetes cluster for while following the official documentation without any success. It turned out the official documentation was missing some important steps (or they put the missing steps else where I couldn’t find). Anyways, if you are struggling to get a Kubernetes up and running, this step by step tutorial is for you.

I’m going to setup a k8s cluster with 1 master node and 1 worker node. Once you have a master node up and running, adding one or more worker nodes does not require extra expertise.

I also use VirtualBox running two identical Ubuntu 18.04 VM. I guess that the newer Ubuntu versions should work fine (haven’t tested).

Step by step to install kubernetes cluster

Here are the steps you need to run on all nodes

Step 1: Disable swap

To disable swap, simply remove the line with swap in /etc/fstab

sudo vim /etc/fstab

Comment out the line with swap

Step 2: Install docker run time

 sudo apt-get update
 sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

 sudo apt-get update
 sudo apt-get install docker-ce docker-ce-cli containerd.io

Step 3: Configure cgroup

Switch to root and run

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

systemctl restart docker

Step 4: Install kubeadm, kubelet, kubectl

sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg

echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

Now, that's all the common commands you need to run on all nodes. Next comes the command you only run on the master node:

Step 5: start master node

kubeadm init

You should see similar message after a few minutes:

Copy the kubeadm join... command to later run on worker nodes.

Finally, you need to install network plugin for the master node (super important!)

sudo kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(sudo kubectl version | base64 | tr -d '\n')"

Wait for a few minutes for the master node to be ready. You can run:

kubectl cluster-info

and wait until the status of the master node is Ready

Step 6: Join the cluster on worker nodes

Then, switch to the worker node and run the join command (the one you got after kubeadm init)

kubeadm join 192.168.1.98:6443 --token 0mfz2s.4xt0waiyfnpxiyt9 \
        --discovery-token-ca-cert-hash sha256:12e48d3bbfb435536618fc293a77950c13ac975fbea934c49c39abe4b7335ce1

Back to the master node and run

watch kubectl get nodes

It will watch the cluster and after a few minutes, you should see all the nodes are ready:

Congratulations! You have successfully setup a kubernetes cluster

Create Lumen development environment in 1 minute

Mạnh Đạt — Tue, 12 Jan 2021 09:11:36 +0000

So recently I need to study lumen (from Laravel). I used Laravel back in version 4.2. A lot of things has changed. I need to setup the environment very quickly and dive right into writing application.

Having not so pleasant setting up Laravel (with Homestead, Vagrant...) I decide to find a method that help me (and other people like me) to get started quickly. After a few hours, I finally have the setup I really like.

TLDR:
Here is the video:
https://www.youtube.com/watch?v=nympC_8CWwE&feature=youtu.be
Here is the git repo:
https://github.com/datmt/docker-microservices

For the patient reader, read on.

Step 0: Clone the repo

This is a must.

Step 1: choose a domain name, put in your hosts file.

Don't worry, you don't need to buy any domain. You can pick any domain, even google.com.

In my case, I used cnn.com.

Step 2: Update the domain in default.conf file

Now, in the lumen-dev folder, there is a file called default.conf. This is the default configuration for nginx server.

Find the part says cnn.com and replace it with your own domain.

This is the content of the default.conf file

server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;

    root   /usr/share/nginx/html;

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass php_server:9000;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
    }
    # concurs with nginx's one
    #
    location ~ /\.ht {
        deny  all;
    }
}


server {
    listen 80 default_server;


    root /usr/share/nginx/html/public;
    index index.php index.html index.htm;

    server_name cnn.com;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
        fastcgi_pass php_server:9000;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
    }


}

Step 3: Install composer packages

cd to src folder and run composer install. In case you don't have composer installed, download the lumen.tar.gz file under archives and extract it. There is a vendor folder there.
Copy that vendor folder to src/.

Step 4: Edit your port

My default configuration use port 8089. If you want to use other port, change it accordingly. One caution if you use port 80 is to make sure that port is free. Many applications use port 80 so you may not be able to start the application.
This is the docker-compose.yml file:

version: '3.8'

services:
    nginx:
        image: nginx:1.19.6
        container_name: nginx
        volumes:
            - ./src:/usr/share/nginx/html
            - ./default.conf:/etc/nginx/conf.d/default.conf
        ports:
            - 8089:80
        links:
            - php_server
    php_server:
        container_name: php_server
        image: php:7.4.14-fpm-buster
        volumes:
            - ./src:/usr/share/nginx/html

Step 5: Use docker-compose to start the app

In lumen-dev folder (where you have docker-compose.yml file), run

docker-compose up -d

Now, you can open your browser and access your application at the port you choose.

Have questions? Please let me know