DEV Community: Watson

Some Notes on Process

Watson — Thu, 03 Sep 2020 13:34:33 +0000

What is process

Process is just a substance of program. Program is a like image containing a set of machine language instructions and some data, which is stored on the disk. You can check processes on your machine using the command "ps". This is the example of the "ps" command running on the WSL of Ubuntu 20.04.

$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 17:23 ?        00:00:00 /init
root         5     1  0 17:23 tty1     00:00:00 /init
watson       6     5  0 17:23 tty1     00:00:00 -bash
watson     111     6  0 19:23 tty1     00:00:00 ps -ef

As you can see, "ps" command is also a process.
All linux distributions have "init" process whose process id is 1. Init process is the first process to run on the machine although the process 0 has already ran in the kernel, to be exact. So all process starts from init process and it is said to be the parents of all processes.

Processes have not only PID but also PPID meaning parents process id. Processed can be represent as a tree structure. In this case, the structure is as below.

There are two inits process because I am using WSL. WSL uses its own init process that is different from linux's. WLS needs to serve 9p server to enble windows to access on the file system of linux. This task is in the same init binary but runs as a different process. That is why there are some init processes when using WSL.

Memory Architecture

This is the overview of virtual memory and executable image. Different UNIX systems use different layouts for processes, but for the most part, most modern systems adhere to a format known as the Executable and Linkable Format (ELF).

Kernel space

Process structure

The kernel maintains process structure for every process. This structure contains the information that the kernel needs to manage the process. The information set it is containing differs among linux distributions or versions but most distributions have the information below.
・Process id
・Parent process id (or pointer to parent's process structure)
・Pointer to list of children of the process
・Process priority for scheduling, statistics about CPU usage and last priority
・Process state
・Signal information (signals pending, signal mask, etc.)
・Machine state
・Timers

User structure

The user structure maintains far less information than the kernel structure. In Linux, it contains the memory maps and the process control block. The memory maps generally include the starting and ending addresses of the text, data, and stack segments, the various base and limit registers for the rest of the address space, and so on. The process control block contains the CPU state and virtual memory state.

Kernel stack

Each process has its own kernel stack. All functions in the kernel space is carefully designed so that they are non-recursive because recursive functions use a lot of stack spaces. The max possible size of stack can be determined by tracing the function chain. So the kernel stack is allocated in a fixed size.

User space

Text segment

The text segment is the programs's executable code. This segment is read-only and shared with any processed.

Stack segment

The stack segment storage for the return address of function, arguments of function and so on. If the stack meets the top of the heap, it causes an exception.

Data segment

Data segment maintains the initialized data and uninitialized data. Initialized data has starting value and its name coming from symbol table. Uninitialized data doesn't have its value so it has only the offset of data segment. The data segment grows or shrinks by explicit memory requests such as brk() system call. malloc() is brk() related function in C.

Type of ELF

There are 3 types of ELF. We can get it with the process of compile.

1 - Object file (*.o)
This is the binary holding code and data for linking with other object files to create executable. This is like a part of executable so it cannot executed by itself.

2 - Executable
This is the binary which can be executed. All object files needed to execute a program are linked and executable is generated. *.a file is a static library which is archive of some object files. When the program needs static library, linker links it when compiling. If you don't define the name of output executable, the name will be a.out meaning assembler output.

3 - Shared object file (*.so)
This is known as a dynamic linking library. When the program needs it, it will be linked when executing.

What Is x86

Watson — Sun, 23 Aug 2020 07:10:16 +0000

Introduction

I hear x86, x86-64, AMD64 and ARM often but I don't know what they mean in detail. We need to know "ISA" to dive into them. Let's start with it.

What Is ISA

ISA is the definition of operations, addressing mode and storage locations. It also describes how to invoke and access them. It works as an interface between software and hardware. Why do we need it?
In the beginning of computer era, each machine had its own instruction set. It means one software runs on one type of machine and we cannot use it in other machines. It was so inconvenient. So IBM started to integrate instruction set and define ISA. Every software can run on every machine if they are according to the same ISA. This reinforced the versatility of software.

There are three type of ISAs in term of storage locations.
・Stack
The number of operands is zero. When the operations require data, it will be manged on the stack.
・Accumulator
This uses memory address as operands.
・General Purpose Register (GPR)
Data is stored in the register and the operations take it as the operands.

Let's think about A+B=C on each architecture. The simple assemblies are as below.

This history and the changes of name confuse us a lot :(

Famous ISAs

x86, x86-64, AMD64 and ARM are the names of ISA. Here is a brief table to know the difference at the glance.

History of ISAs for PC

Actually, x86-64 and AMD64 are the same things. The simple history of ISA in PC is as below
・Intel developed 80386 which was a first 32 bits CPU. Other series were released and they came to be called "x86" generically
・Intel developed 64 bits CPU IA-64 architecture but it didn't spread because it was not compatible with x86.
・AMD developed 64 bits CPU x86-64 architecture and it became popular. It changed the name to AMD64 after a few years.
・Intel developed 64 bits CPU IA-32e architecture and changed the name to EM64T and to Intel64. It is compatible with AMD64.

This history and the changes of name have confused us a lot :(

The type of architecture

・CISC(Complex Instruction Set Computing)
The task on the desktop PC tends to be complicated. CISC architecture is suitable for it. The instruction set consists of complex commands which can be divided into smaller sub commands and whose size is different. When the processor uses commands, they are sent to the decode unit and translated as a microcode. Nano processor(like a processor in a processor) processes it.

・RISC
The task on the mobile tends to be simpler than PC. RISC is suitable for this case. The instruction set consists of simple commands whose size is same. There is no overhead to divide the commands into pieces so that they are only needed to be decoded to micro codes. This architecture enables it to read commands and process it faster.

Register

The big difference is an available memory size. When programs need to access to the memory, the required address loads into the register. So, the limitation of the size program can handle depends on the size of the register. For example, 32 bit CPU can handle about 40 billion addressed, which means 4GB and 64 bit CPU can handle 16EiB.

P.S. What else should I know to understand deeply of computer architecture or CPU??
Enjoy hacking!!

What Is a difference between HDD and SSD

Watson — Sun, 16 Aug 2020 08:37:12 +0000

Permanent storage is one of the most essential part in the modern computer. All data in the computer disappears when the power turns off without permanent storage.
We often use HDD or SSD and SSD is more useful and expensive one. Why does SSD perform better? Let's take a close look at them.

The System of HDD

HDD is just a computer by itself because it has control units, a power socket and memory. There are 4 main parts in HDD.

1.Platter

Platter is a magnetic disk, which is covered with a lot of tiny magnetized metal grains. Magnet generates "magnetic field" which has the power towards one direction. This direction represents a bit like upward means '0' and downward means '1'.

We can determine the data's physical position with cylinder, sector and track.

Cylinder

This denotes which plate and side will be used.

Sector

This denotes which part of plate on the surface will be used. Platter is divided every 512 (B) in some HDD, and 4 (KB) in others. There are non-recording areas among sectors. So smaller sector is not good because it generates many non-recording areas.

Track

This denotes how long the distance from center is.

2.Spindle

Spindle spins the platter to set the sector position of requested data. RPM means the number of rotations in one minute. Major HDD is 5400(RPM) or 7400(RPM) and we can access data in shorter time when RPM gets larger.

3.Head

Head reads and writes data on the platter. It has a strong magnet and can change the direction of magnetic field. This means it has an ability to write data. Also it can know the direction of magnetic field and convert it to bit.

Head is flying nano meters above the platter not to rub the platter.

4.Actuator

Actuator is main part that moves the head. When HDD is not working, head should be placed on other than the platter. HDD is divided two types by the palce head is on when HDD is not working.

Lamp load type requires another part but it has stronger resistance to shock.

The System of SSD

SSD has a control unit and a cache memory as well but it doesn't have platters and a head. When SSD reads or writes data, it doesn't need to move physical parts. This is one reason that SSD performs better. Also the way of recording data is completely different from HDD. This makes the physical size of SSD smaller. Here is the architecture.

1.Flash Memory

Flash memory records data using floating gate transistors. Floating gate with electronic means 0 and one without electronic means 1. This is called SLC(Single Level Cell). We can represent 4 types of value, which means 2 bits, to change the amount of electronic like none, small, medium and large. This is called MLC(Multi Level Cell). It helps SSD to have more storage capacity.

2.DRAM

This works as a cache and helps user with efficient access to data.

3.Controller

Controller has two important roles.
The first one is wear leveling. Each flash memory cell has lifetime and it is not good to read or write data in the same cell. So, Controller memorizes the number of use in each cell and control the access from user to increase the lifetime of memory.
The second is error correction. The older the cell is, the more it generates error. Controller attaches some data to correct error when it writes some data in the cell.

Enjoy hacking!

I Promise Your Deep Understanding of Promise

Watson — Wed, 29 Jul 2020 11:14:03 +0000

TL;DR

I describe background system of asynchronous function in Javascript and how to use promise a little.

Introduction

You heard that Javascript is single thread and asynchronous model so many times. But we can fetch the data from the server while computing some data or event. Someone mistakenly believe that multi threading enables it but it is not true. Asynchrony supports modern Javascript behavior. Let's take a closer look of the asynchronous system and go on the topic of Promise.

What Is Asynchrony in Javascript

First, we need to define the asynchrony in Javascript. I think there are three keys for defining as below.

"The program runs from top to bottom along the written code¹. When the function using external resources (WebAPI, Network, Database) is called², the program will not wait for the return of the function and run next code³."

This behavior is necessary not to idle the CPU. CPU should do other important works such as local computation or rendering while using external resources. So asynchrony improves efficiency although the programming model would be a little complicated.

The program including "setTimeout" function is one of the famous examples running asynchronously. This example is often used because we should call the function using out resources to let the program behave asynchronously and it is very simple.
You know, "setTimeout" function is just an interface and a browser actually counts times.

==Simple Example==

console.log("1");
setTimeout(function() {
    console.log("2");
}, 1000);
console.log("3");

// output
// 1
// 3
// 2

You can understand the result intuitively because console.log("2") runs after 1000(ms) and console.log("3") have already ran before that.

==Counterintuitive Example==

console.log("1");
setTimeout(function() {
    console.log("2");
}, 0);
console.log("3");
// output
// 1
// 3
// 2

The result is same as previous one although console.log(2) waits 0(ms).
We need to understand the back system of calling function to know why this happened.
Loupe helps us a lot to see the flow of calling function.
The overview is something like as below.

The important thing is that callback functions in the queue will not run until the call stack is empty. This is the non-blocking feature.

Generally we can say :

The function is first registered in the call stack
When the function uses external resources, the callback function is registered in the queue
Event loop always monitors stack and if the stack is empty, it places one of the call back functions on the stack (In fact the runtime is multi threading)

What Is The Problem without Promise

Long story short, the problem is "callback hell". If you want to run some asynchronous functions serially, you should write the next processing in the callback function of previous one.
We can easily understand with a simple example.

==Situation==
We would like to read four files (A.txt, B.txt, C.txt, D.txt) whose contents are A,B,C,D respectively and catenate them in order like ABCD.

If you are not familiar with asynchronous functions, this code can be written.

const fs = require("fs");
let all = "";

fs.readFile("A.txt", function (err, data) {
    all += data;
});

fs.readFile("B.txt", function (err, data) {
    all += data;
});

fs.readFile("C.txt", function (err, data) {
    all += data;
});

fs.readFile("D.txt", function (err, data) {
    all += data;
});

setTimeout(function () {
    console.log(all);
}, 100);

// Outputs of some runnings
// ABDC
// ABCD
// ADCB
// ABDC
// ABCD

We can get "ABCD" sometimes, but you cannot get "ABCD" certainly every time. The functions are placed on the call stack in order but the I/O time varies even if it reads same file so that the order of registering the callback function to the queue is different from the one of placing on the call stack.

Now we know that it will work correctly when the callback function is registered in the queue in order. So the way the next computation step is in the previous callback function sounds good.

const fs = require("fs");
let all = "";
fs.readFile("A.txt", function (err, data) {
  all += data;
  fs.readFile("B.txt", function (err, data) {
    all += data;
    fs.readFile("C.txt", function (err, data) {
      all += data;
      fs.readFile("D.txt", function (err, data) {
          all += data;
          console.log(all);
      });
    });
  });
});
// Outputs of some runnings
// ABCD
// ABCD
// ABCD
// ABCD

We can get "ABCD" every time as expected because the code runs repeatedly as below.

As you can see, the code is nested more deeply if the length of callback function chain gets longer. This is called, as mentioned above, "Callback Hell". It's hard to understand and maintain such a code. Promise solves this problem.

What Is Promise about

It's natural that we want to handle asynchronous behavior just like other functions, which return some objects after processing. Promise enables us this feature.

To be simply put, promise is like a intermediary between javascript world and external resources world. Promise guarantees that it will get the result from external resources in the future. So, you can ask everything about external resources such as return values or error codes to promise, don't you think he is a great partner, man?

Promise has three states.

Pending

This is an initial state and promise is waiting for the response from external resources.

Fulfilled

This denotes that promise have already known that external resources succeeded in the process and got some return values from external resources.

Rejected

This denotes that promise have already known that something wrong happened in external resources and got the reason from external resources.

We can see the three states in the simple example.

const fs = require("fs").promises;

// Pendding : Print as soon as promise object is created
let promise1 = fs.readFile("A.txt");
console.log(promise1);

// output
// Promise { <pending> }

// Fullfilled : Print after a second
let promise2 = fs.readFile("A.txt");
setTimeout(function () {
  console.log(promise2);
}, 1000)

// output
// Promise { <Buffer 41> } 
// ↑ Promise has tha returned Buffer object. 0x41 means 'A'.

// Rejected : Read not existing file (E.txt)
let promise3 = fs.readFile("E.txt");
setTimeout(function () {
  console.log(promise3);
}, 1000)

// output
// Promise {
//   <rejected> [Error: ENOENT: no such file or directory, open 'E.txt'] {
//     errno: -2,
//     code: 'ENOENT',
//     syscall: 'open',
//     path: 'E.txt'
//   }
// }

We often use the phrase "if ~, then ~". We can think about Promise like "If the value return from external resources, then do something with it". Anyone doesn't know whether the function will succeeds in processing or fail but we can just write a code for the future. So a promise object prepares "then" function. We write the processing for the future success in "then" function and for the fail in "catch" function. Be careful the fact that the processing in "then" and "catch" will be just registered in the queue and not run immediately.

const fs = require("fs").promises;
let promise = fs.readFile("A.txt");
promise
  .then((data) => {
    console.log(data.toString());
  })
  .catch((err) => {
    console.log(err);
  });

// Generalization
// (Promise Object)
// .then((the returned value) => do something)
// .catch ((the reason of error) => do something)
// .finally(() => do something in both cases )

Promises Chain

We know the fundamental things of promise. But we cannot solve the "ABCD" problem without deep nest right now.
As we can imagine, this code doesn't work.

const fs = require("fs").promises;
let all = "";

fs.readFile("A.txt").then(data => {
  all += data.toString();
});
fs.readFile("B.txt").then(data => {
  all += data.toString();
});
fs.readFile("C.txt").then(data => {
  all += data.toString();
});
fs.readFile("D.txt").then(data => {
  all += data.toString();
});
setTimeout(() => {
  console.log(all);
}, 1000)

// outputs
// ABCD
// ABCD
// ACBD
// CBAD
// BCAD

The file reading functions run in order but the processing in the "then" function will be registered when the I/O finishes so the timing varies every time. This is just a image for helping you understand.

In this situation, promises chain helps us.
Actually, "then" function returns a promise object. When we don't specify the promise object, it returns default undefined promise object. We return the next promise object in the previous "then" function so that the promises can be processed serially.

fs.readFile("A.txt")
  .then((data) => {
    all += data.toString();
    return fs.readFile("B.txt");
  })
  .then((data) => {
    all += data.toString();
    return fs.readFile("C.txt");
  })
  .then((data) => {
    all += data.toString();
    return fs.readFile("D.txt");
  })
  .then((data) => {
    all += data.toString();
    console.log(all);
  });
// outputs
// ABCD
// ABCD
// ABCD
// ABCD

This is the promises chain and it's really easy to read and understand!! And here is a same image as previous.

Other Tips

We can use Promise.all() to solve the "ABCD" problem. Promise.all receives some Promise objects and we can think them as if they were a single Promise object and would return all values at once.

const fs = require("fs").promises;
Promise.all([
  fs.readFile("A.txt"),
  fs.readFile("B.txt"),
  fs.readFile("C.txt"),
  fs.readFile("D.txt"),
]).then((values) => {
  console.log(values);
});
// output
// ABCD

Thank you very much for reading this through to the end!!
Enjoy hacking!!

The Flexible Flamework When I Explain A Technology

Watson — Fri, 24 Jul 2020 09:14:34 +0000

Hey guys.
I would like to post more thchnical stuffs.
What do you think what are important things to write on the post when you explain some technical things ?

My framework is as below.

Background of the technology and what is problem in that field
What is technology about
How the technology solves the problem
Application to real problems
Other tips
What is existing problem and what are future works I am expecting

Your opinion will meke my post better and better.
Enjoy hacking!

How to Send Pair with Open MPI

Watson — Fri, 24 Jul 2020 07:24:07 +0000

Hey guys.
I'm a beginner of Open MPI.
When I wanted to send some std::pair onjects from a node to another, I found out the fact that there are no Datatype of std::pair.
You can use only the datatypes in Open MPI as bellow.

MPI Datatype	C++ Datatype
MPI::CHAR	char
MPI::SHORT	signed short
MPI::INT	signed int
MPI::LONG	signed long
MPI::LONG_LONG	signed long long
MPI::SIGNED_CHAR	signed char
MPI::UNSIGNED_CHAR	unsigned char
MPI::UNSIGNED_SHORT	unsigned short
MPI::UNSIGNED	unsigned int
MPI::UNSIGNED_LONG	unsigned long int
MPI::UNSIGNED_LONG_LONG	unsigned long long
MPI::FLOAT	float
MPI::DOUBLE	double
MPI::LONG_DOUBLE	long double
MPI::BOOL	bool
MPI::COMPLEX	Complex
MPI::DOUBLE_COMPLEX	Complex
MPI::LONG_DOUBLE_COMPLEX	Complex
MPI::WCHAR	wchar_t
MPI::BYTE
MPI::PACKED

You can see only basic types and it's natural to do so because developers use variety of custom types and Open MPI can't prepare for all types of them.

I think the most flexible one is MPI::BYTE.
So, I used it to send std::pair.
You can easly calculate the total size of pairs in 1 byte increments by using the "sizeof" function.

Here is the example code.

#include "mpi.h"
#include <bits/stdc++.h>
using namespace std;
int main(int argc, char **argv) {
  int rank, numprocs;
  int namelen;
  vector<pair<int, int>> send_buf = {make_pair(0, 1), make_pair(2, 3)};
  vector<pair<int, int>> recv_buf(2);

  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
  if(rank == 0){
    MPI_Send((void*)send_buf.data(), sizeof(pair<int,int>)*2, MPI_BYTE, 1, 0, MPI_COMM_WORLD);
  }

  if(rank == 1){
     MPI_Recv((void*)recv_buf.data(), sizeof(pair<int,int>)*2, MPI_BYTE, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
     cout << recv_buf[0].first << ":" << recv_buf[0].second << endl;
     cout << recv_buf[1].first << ":" << recv_buf[1].second << endl;
  }
  MPI_Finalize();
  return 0;
}

Enjoy hacking!