DEV Community: Taher Fattahi Tabalvandan

Markov Decision Processes(MDP) basic concept

Taher Fattahi Tabalvandan — Thu, 23 Nov 2023 13:31:46 +0000

A Markov decision process (MDP) is a mathematical framework used to model decision-making problems in which outcomes are partly random and partly under the control of a decision-maker. It consists of a set of states, a set of actions, and a set of rewards. The decision-maker takes actions in each state, and the environment transitions to a new state based on the action taken and a probability distribution over the next state. The decision-maker receives a reward based on the state and action taken.

Here’s an example of an MDP:

Consider a robot that is trying to navigate through a maze. The robot can move in four directions: up, down, left, or right. The robot’s goal is to reach the exit of the maze. The robot’s current state is its current location in the maze. The robot can only see a limited area around it, so it doesn’t have access to the entire maze. The robot’s actions are to move in one of the four directions. The environment transitions to a new state based on the action taken and a probability distribution over the next state. For example, if the robot moves up, there might be a 70% chance that it stays in the same location, a 10% chance that it moves to the left, a 10% chance that it moves to the right, and a 10% chance that it moves down. The robot receives a reward based on the state and action taken. For example, the robot might receive a reward of +10 if it reaches the exit of the maze, a reward of -1 if it bumps into a wall, and a reward of -0.1 for each step it takes.

The environment is typically formulated as a Markov Decision Process

1.) set of states: S
In chess, a state is a given configuration of the board

2.) set of actions: A (for example: up, down, left, right)
Every possible move in chess or tic-tac-toe

3.) Conditional distribution of the next state
The next state depends on the actual one exclusively „Markov-property"
P(s'|s,a) transition probability matrix

4.) R(s,s') the reward function of the next s' state after the agent makes action a while being in the state s

5.) γ gamma discount factor

The Markov property is a fundamental assumption in reinforcement learning. It states that the current state of an agent contains all the information that is relevant for decision-making. In other words, the future state of the agent depends only on its current state and not on its past states. This assumption enables theoretical proofs for certain algorithms and is widely used in current research.

For example, in a Markov decision process (MDP), we assume that the current state is independent of the whole history given the previous state. However, in real-life scenarios, this assumption may not always hold true and can lead to an RL algorithm failing in a specific environment.

Reference:
https://web.stanford.edu/class/cme241/lecture_slides/david_silver_slides/MDP.pdf

FluffyChat push notification with sygnal!!

Taher Fattahi Tabalvandan — Thu, 04 Aug 2022 07:30:00 +0000

How you can add FluffyChat messenger to your custom application with your package name or your unique bundle id?

FluffyChat source code => Link

In my new project development, I realized some information about FluffyChat Messenger.

You can add all of the FluffyChat source code to your new project but absolutely your push notification not working,
if you want to use Firebase Cloud Messaging in FluffyChat for sending a push notification in your new application.

Plz follow these steps:

1) Create new app in https://console.firebase.google.com/ and add google-serices.json to your project

2) Setup and install sygnal in your server

3) In FluffyChat source code plz change pushNotificationsAppId and pushNotificationsGatewayUrl in app_config.dart file

4) Add this line of code in sygnal source code:
in gcmpushkin.py add bottom line of code to _build_data function:

data["devices"] = [{"pushkey": device.pushkey,"data": device.data,"pushkey_ts": device.pushkey_ts,"app_id": device.app_id}]

You can see this repository:
https://github.com/taherfattahi/sygnal

5) Add FCM Server key to sygnal.yaml in sygnal project

If you have any problem you can get in touch with me, my email address: taherfattahi11@gmail.com 😉

Build a Chat service using GoLang and WebAssembly (part 4)

Taher Fattahi Tabalvandan — Sat, 30 Jul 2022 15:32:00 +0000

ok, for communicating with server we need a Websocket Server - i prefer we develop our WebSocket server-side with nodejs.

our source code => Github Link

What is WebSocket?
Websocket is a simple to use, fast and tested WebSocket client and server implementation.

The Websocket specification defines an API establishing a connection between a web browser and server. WebSocket is used for creating real-time games, chat applications, displaying stock data, etc.

NodeJS and WebSocket
NodeJS is a JavaScript runtime built on Chrome’s V8 JavaScript engine. It is easy to build a WebSocket server implementation with NodeJS.

Traditional HTTP is dependent on client requests, you have to send your request to the server and then get a response from it but in WebSocket, you can send data to the client from the server directly which we call bidirectionally.

Let me introduce you to Websockets: An event-driven, web-friendly alternative to HTTP. Websockets don't require a client request to fetch data from the server every time.

Building a WebSocket Server With NodeJS
As prerequisites, you should have Nodejs and NPM Installed in your system. If you don’t have it, download and install it from here: https://nodejs.org/en/download/

1) Once you are ready with NodeJs, open a terminal and type the following commands:

mkdir webSocketServer && cd webSocketServer

2) Create your nodejs project environment:

npm init -y

3) Next, run the following command to install the ws library as a dependency:

npm install ws

4) Create a javascript file for example “webSocketServer.js” and paste the following code for creating a web server:

// Importing the required modules
const WebSocketServer = require('ws');

// Creating a new websocket server
const wss = new WebSocketServer.Server({ port: 8080 })

// Creating connection using websocket
wss.on("connection", ws => {
    console.log("new client connected");
    // sending message
    ws.on("message", data => {
        console.log(`Client has sent us: ${data}`)
    });
    // handling what to do when clients disconnects from server
    ws.on("close", () => {
        console.log("the client has connected");
    });
    // handling client connection error
    ws.onerror = function () {
        console.log("Some Error occurred")
    }
});
console.log("The WebSocket server is running on port 8080");

This code will create a basic WebSocket Server for you and you can provide any port you want while creating the WebSocket server.

5) For testing it, open up a terminal and type:

node webSocketServer.js

Reference:
1) piesocket.com

In part 5 we are going to write our WebSocket client-side service with Golang WebAssembly and connect to WebSocket nodejs server-side😉

Build a Chat service using GoLang and WebAssembly (part 3)

Taher Fattahi Tabalvandan — Wed, 20 Jul 2022 06:14:16 +0000

Gooood :)))) now we are in part 3 - we wanna talk about Chat Service Architecture

A chat service consists of two major parts:

Chat App or client part, which is a desktop, web, or smartphone chat application.
Chat Server Engine or server part, which is a pool of external servers responsible for the chat operation. This is the place where all the chat magic happens.

Both parts contain various components that communicate to each other and bring the chat into action.

Chat Server Engine is a core of the chat architecture that handles message delivery and dispatch. I want to show you all components in chat service:

Chat REST API handles the tasks that are not connected directly to message dispatch and delivery, such as user authentication, changing of user settings, friends invitation, downloading sticker packs, etc. The Chat App (the chat client part) communicates with the Chat REST API via the Chat REST API Client Library.
Chat WebSocket Server is responsible for transmitting messages between users. The Chat App communicates with the Chat WebSocket Server via the Chat WebSocket Client Library. This connection is open two ways; that means users don’t have to make requests to the server if there are any messages for them, they just get them right away.
Chat Media Storage Server is a pool of servers responsible for storing user media files. The files are uploaded to them via the Chat Media Storage Client Library.

Chat App is the other major part of the chat architecture, the one that users directly interact with. It's split into three separate root components:

Chat Client Engine handles all of the communication with the Chat Server Engine via its internal components: Chat REST API Client Library, Chat WebSocket Client Library and Chat Media Storage Client Library. It also comprises the Chat Push Message Handler that deals with push notifications.
Chat UI displays data to users via its widgets: Chat Contact List UI, Chat Dialog UI, Chat Push Message Widget — extension for mobile apps that allow for replying to messages without opening the app and Chat Internal Notification Widget — a widget that pops up at the top of the screen while the user is chatting in a dialog and notifies about the incoming message in another dialog.
Chat Device Storage is an internal database (read: your device storage), which stores messages and files so that users can access them offline. Its internal component, Chat Media Cache, gets media files from the Chat Media Storage and stores them on the device so that the user can access them anytime without having to reach the Chat Media Storage every time.

So we need WebSocket protocol that makes it possible to open a two-way interactive communication session between the user's browser and a server. so you can send messages to a server and receive event-driven responses without having to poll the server for a reply.

In our project we create a websocket server with nodejs and a websocket client with Golang

References:
1) yellow.systems: Chat Service Architecture
2) developer.mozilla.org: WebSocket

In part 4 we are going to write our websocket server with nodejs😉

Build a Chat service using GoLang and WebAssembly (part 2)

Taher Fattahi Tabalvandan — Fri, 15 Jul 2022 09:58:00 +0000

ok, now we want to write our first WebAssembly code with golang

this is our source code structure:

our source code => Github Link

Hello World WebAssembly Program Cross Compiled from Go:

Let's start by writing a simple hello world program in Go, cross compile it to WebAssembly, and run it on the browser.

1) At the first you have to install golang compiler on your device => golang

2) Creating a directory with a new project:

mkdir golangChatWebAssembly 
cd golangChatWebAssembly

3) Let’s create golangChatWebAssembly module:

go mod init golangChatWebAssembly

4) Let’s create main.go file

mkdir app && cd app
touch main.go

5) Let’s fill our application with content

app/main.go

package main

import (  
    "fmt"
)

func main() {  
    fmt.Println("Go Web Assembly")
}

6) Now it’s time to build our app

This command will compile our go application into the main file. wasm (with wasm bytecode) is an executable module of a web assembly.

if you have Linux or MacOS use bottom command:

GOOS=js GOARCH=wasm go build -o ./html/main.wasm .

if you have Windows OS I suggest creating build.bat file and then running it:

touch build.bat

on your build.bat file write this syntax:

SET GOOS=js
SET GOARCH=wasm

go build -o ./html/main.wasm .

SET GOOS=windows
SET GOARCH=amd64

and then run it in command line: .\build.bat

7) Javascript Glue

As we already discussed (part 1), WebAssembly is supposed to exist hand in hand with JavaScript. Hence some JavaScript glue code is needed to import the WebAssembly Module we just created and run it in the browser. This code is already available in the Go installation. Let's go ahead and copy it to our html directory.

cp "$(go env GOROOT)/misc/wasm/wasm_exec.js" ./html/

The above command copies the wasm_exec.js that contains the glue code to run WebAssembly into the html directory.

in Windows OS download wasm_exec.js from this link and then copy file into html folder

8) we have to create a HTML page with an application call to the Web Assembly

create index.html in app/html directory:

<html>
    <head>
        <meta charset="utf-8"/>
        <script src="wasm_exec.js"></script>
        <script>
            if (!WebAssembly.instantiateStreaming) {
                WebAssembly.instantiateStreaming = async (resp, importObject) => {
                    const source = await (await resp).arrayBuffer();
                    return await WebAssembly.instantiate(source, importObject);
                };
            }

            const go = new Go();
            WebAssembly.instantiateStreaming(fetch("main.wasm"), go.importObject).then(result => {
                go.run(result.instance);
            });
        </script>
    </head>
</html>

9) we will also need a web server

Let’s create a simple web server on Go

cd ..
mkdir server && cd server
touch server.go

fill our server.go with this source code:

package main

import (
    "log"
    "net/http"
)

const (
    AddSrv       = ":8080"
    TemplatesDir = "../app/html/"
)

func main() {
    fileSrv := http.FileServer(http.Dir(TemplatesDir))

    err := http.ListenAndServe(AddSrv, fileSrv)

    if err != nil {
        log.Fatalln(err)
    }
}

10) After that, we start the server

server/

go run server.go

and open the address in the browser

http://127.0.0.1:8080

and in “DevTools > console” you can see the output:

Go Web Assembly

In part 3 we are going to study about chat service architecture 😉

Build a Chat service using GoLang and WebAssembly (part 1)

Taher Fattahi Tabalvandan — Thu, 14 Jul 2022 08:23:51 +0000

At the first I want to explain what is Webassembly?

in short, compile and run languages such as java-c#-python,.... into the browser we called Webassembly.

JavaScript has been the only programming language that the browser understands. JavaScript has stood the test of time and it has been able to deliver the performance needed by most web applications. But when it comes to 3D games, VR, AR, and image editing apps, JavaScript is not quite up to the mark since it is interpreted. Although JavaScript engines such as Gecko and V8 have Just in Time compilation capabilities, JavaScript is not able to provide the high performance required by modern web applications.

WebAssembly(also known as wasm) is meant to solve this problem. WebAssembly is a virtual assembly language for the browser. When we say virtual, it means that it cannot be run natively on the underlying hardware. Since the browser can be running on any architecture, it is not possible for the browser to run WebAssembly directly on the underlying hardware. But this highly optimized virtual assembly format can be processed much quicker than vanilla JavaScript by modern browsers since it is compiled and is more close to the hardware architecture than JavaScript. The following figure shows where WebAssembly stands in the stack when compared to Javascript. It is closer to the Hardware than JavaScript.

The existing JavaScript engines have support to run WebAssembly's Virtual Assembly Code.

WebAssembly is not meant to replace JavaScript. It is meant to operate hand in hand with JavaScript to take care of performance critical components of a web application. It is possible to make calls from JavaScript to WebAssembly and vice versa.

WebAssembly is not generally coded by hand, but rather, it is cross compiled from other high level programming languages. For example, it is possible to cross compile Go, C, C++, and Rust code to WebAssembly. Thus the module which has already been coded in some other programming language can be cross compiled to WebAssembly and used in the browser directly.

Application of WebAssembly (WASM)

Where was WebAssembly supposed to be used at first? First of all, in tasks for processing heavy-duty tasks directly in the browser.

Game development
Video editing/streaming
Encryption
Image recognition

In part 2 we are going to write our first WebAssembly code with golang 😉