DEV Community: Petro Liashchynskyi

NestJS CODEX: auxiliaries for CRUD using Mongoose with transactions support

Petro Liashchynskyi — Sat, 04 Nov 2023 09:45:00 +0000

I've been working with Nest for quite a long. Setting up base stuff such as CRUD services, etc. is a bit of exhausting. So I've created a project that can help you with that.

Repository provides a robust CRUD service using NestJS and Mongoose, designed to simplify the development of database interactions with built-in transaction support via Async Local Storage.

Features

CRUD Operations: Simplify create, read, update, and delete operations using Mongoose.
Transaction Management: Handle transactions smoothly and reliably in your services.
Async Local Storage: Utilize Async Local Storage for context management throughout the life of a request.

For more information please refer to https://github.com/liashchynskyi/nestjs-codex/

This post was originally posted on my blog - https://liashchynskyi.net/posts/nestjs-codex-auxiliaries-for-crud-using-mongoose-with-transactions-support

Sending posts from WordPress site to your Telegram channel

Petro Liashchynskyi — Sat, 08 Feb 2020 11:20:45 +0000

Hi! I'm gonna show you how you can send a message (WordPress post) to a Telegram channel. All you need is a public Telegram channel and a bot. The last one you can create via BotFather in Telegram.

Steps:

Create a public channel and a Telegram BOT (via BotFather).
Remember the bot token.
Add the bot to administrators of previously created channel.

At the final step you should add the following code to your functions.php:


function telegram_send_message( $new_status, $old_status, $post ) {
  if( $new_status == 'publish' &&  $old_status != 'publish' && $post->post_type == 'post') {
    $apiToken = "TOKEN";
    $data = [
      'chat_id' => '@channel_name',
      'text' => "\nRead more: " . get_permalink($post->ID)
    ];
   $response = file_get_contents("https://api.telegram.org/bot$apiToken/sendMessage?" . http_build_query($data) );
  }
}
add_action( 'transition_post_status', 'telegram_send_message', 10, 3 );

Note: Do remember to replace TOKEN with your actual bot token and channel_name with the name of the channel.

Now, all the new posts will be also published in your Telegram channel.

My first and not very successful experience with AWS

Petro Liashchynskyi — Tue, 19 Nov 2019 18:42:06 +0000

Hello, today I'm gonna tell you a story about my experience with Amazon Web Services.

I started using it not so long time ago. What is AWS? Amazon provides on-demand cloud computing platform. VM instances, databases, load balancers, and more. And it's so cool 😁

So if you create an account for the first time you have the ability of 12 months "free" using several AWS services. This thing is called Free Tier and restricts the use of certain services. For example, the using of EC2 (virtual machines) service is limited to 750 hours. You will not be charged for 750 hours of using your VM and that's it.

Unfortunately, there's is a thing I didn't know about when using EC2.

Standard vs Unlimited

If you create an EC2 instance of t2 or t3 type, then there is a thing enabled by default which called unlimited feature. And it's bad in my case 💩

Ok, let's figure it out. Every EC2 VM has its own bound of CPU utilization called baseline. Imagine, you are using a VM for heavy tasks when the CPU is loaded above baseline, even at 100%. In that case (if you have unlimited feature enabled), you'll pay money for extra CPU utilization.

But... If you have standard feature enabled, then your CPU utilization will not exceeds baseline bound. Meanwhile, if the CPU frequency rises above the baseline, it will immediately be reduced - great, because there is no additional cost.

Unfortunately, I didn't know that and paid almost 9 dollars. Despite the fact that I currently used only 552 hours of 750 free and the average workload per month.., attention, is zero. I don't know how the costs is computed on AWS 😆 I don't know why they charged me $9. But the money was refunded to me. Thanks to Melanie from AWS tech support ♥️.

Thoughts

When creating VMs in EC2 - please, disable unlimited feature if you really don't need it enabled.

EC2 -> Instances -> Actions -> Instance Settings -> Change T2/T3 Unlimited -> Disable

That's it. Hope it helps someone of you 😉 Stay awesome! Thanks!

Small, fast and simple Python CLI image converter for CNNs

Petro Liashchynskyi — Tue, 30 Jul 2019 15:34:32 +0000

Hello, people) I've been working on CLI tool that's gonna help with dataset augmentation and image converting for CNNs or GANs or even other thing that needs images as input data.

Here it is https://github.com/liashchynskyi/rudi

Support the repo with a star, if you like it! Thanks, hope this tool will help you 😀

Intro to CUDA technology

Petro Liashchynskyi — Wed, 19 Jun 2019 15:04:05 +0000

Hello again! Let's talk about CUDA and how it's gonna help you to speed up the data processing.

No code today! Only theory 😎

Imagine, if you had known there will be no CUDA in the world you would still train any of neural networks forever 🙁 So, what the heck is CUDA?

Intro

CUDA is a parallel computing platform and application programming interface (API) model created by Nvidia (source).

Before we begin, you should understand what is:

device - video card by itself, GPU - runs commands received from CPU
host - central processor (CPU) - runs certain tasks on device, allocates memory, etc.
kernel - function (task) that will be ran by device.

CUDA allows you to implement algorithms using special syntax of C language. The architecture of CUDA let you organize GPU instructions access and manage its memory. All in your hands, bro! Be careful.

Good news - this technology is supported by several languages. Choose the best one 😉

Magic? No 😮

Let's find out how code is launched by GPU.

host allocates some memory on device;
host copies the data from its own memory to device's memory;
host runs kernel on device;
device performs that kernel;
host copies results from device's memory to own memory.

There is no step 1 (allocating memory) on figure, but steps 1 and 2 can be merged.

CPU interacts with GPU over the CUDA Runtime API, CUDA Driver API and CUDA Libraries. The main difference between Runtime and Driver API is pretty simple - it's a level of abstraction.

Runtime API (RAPI) is more abstract, aka more user-friendly. Driver API (DAPI) - a low level API, driver level. In general, RAPI is an abstract wrapper over the DAPI. You can use both of them. I can tell you from my experience it's more difficult to use DAPI, because you should think about low-level things, that's not funny 😑.

And you should understand another thing:

If the time spent on creating the kernel will be greater than the time of this kernel actually running, you'll get zero efficiency.

Anyways, the thing is launching tasks, allocating memory on GPU takes some time, therefore you shouldn't run "easy" tasks on it. Easy tasks can be performed by your CPU in milliseconds.

Should you run a kernel on the GPU even if the CPU can compute it more quickly? Actually no... Why? Let's find it out!

Hardware

The architecture of the GPU has built a bit differently than the CPU. Since graphic processors were originally used only for graphical calculations involving independent parallel data processing, the GPU is designed for parallel computing.

The GPU is designed to handle a large number of threads (elementary parallel processes).

As you can see, the GPU consists of many ALU merged in several groups with common memory. This approach can speed up productivity, but sometimes it's hard to program something in that way.

In order to achieve the best acceleration, you must think about the strategy of memory accessing and take into account the GPU features.

GPU is oriented for heavy tasks with large volumes of data and consists of streaming processor array (SPA), that includes texture processor clusters (TPC). TPC consists of a set of streaming multiprocessors (SM), each of them includes several streaming processors (SP) or cores (modern GPU can have more than 1024 cores).

GPU cores work by SIMD principle, but a bit different.

SP can work with different data, but they should execute the same command at the same moment of time. Different threads execute the same command.

As a result, the GPU actually became a device that implements a stream computing model - there are streams of input and output data, which consist of identical elements, which can be processed independently of each other.

Compute capabilities

Every single GPU has its own coefficient of productivity or compute capabilities - quantitative characteristic of the performance speed of certain operations on the graphic processor. Nvidia called that Compute Capability Version. Higher version is better than lower 😁

Compute Capability Version	GPU Chip	Videocard
1.0	G80, G92, G92b, G94, G94b	GeForce 8800GTX/Ultra, Tesla C/D/S870, FX4/5600, 360M, GT 420
1.1	G86, G84, G98, G96, G96b, G94, G94b, G92, G92b	GeForce 8400GS/GT, 8600GT/GTS, 8800GT/GTS, 9400GT, 9600 GSO, 9600GT, 9800GTX/GX2, 9800GT, GTS 250, GT 120/30/40, FX 4/570, 3/580, 17/18/3700, 4700x2, 1xxM, 32/370M, 3/5/770M, 16/17/27/28/36/37/3800M, NVS420/50
1.2	GT218, GT216, GT215	GeForce 210, GT 220/40, FX380 LP, 1800M, 370/380M, NVS 2/3100M
1.3	GT200, GT200b	GeForce GTX 260, GTX 275, GTX 280, GTX 285, GTX 295, Tesla C/M1060, S1070, Quadro CX, FX 3/4/5800
2.0	GF100, GF110	GeForce (GF100) GTX 465, GTX 470, GTX 480, Tesla C2050, C2070, S/M2050/70, Quadro Plex 7000, Quadro 4000, 5000, 6000, GeForce (GF110) GTX 560 TI 448, GTX570, GTX580, GTX590
........	.........	........
5.0	GM107, GM108	GeForce GTX 750 Ti, GeForce GTX 750, GeForce GTX 860M, GeForce GTX 850M, GeForce 840M, GeForce 830M
........	.........	........

You can find the whole list here. Compute Capability Version describes a lot of parameters such as quantity of threads per block, max number of threads and blocks, size of warp and more.

Threads, blocks and grids

CUDA uses a lot of separate threads for computing. All of them are grouped in hierarchy like that - grid / block / thread.

Blocks struct

The top layer – grid – is related to the kernel and unites all threads performing that kernel. Grid is an 1D- or 2D-array of blocks. Each block is an 1D / 2D / 3D array of threads. In this case, each block represents a completely independent set of coordinated threads. Threads from different blocks cannot interact with each other.

Above, I mentioned the difference from the SIMD architecture. There is still a concept like warp - a group of 32 threads (depending on the architecture of the GPU, but almost always 32). So, only threads within the same group (warp) can be physically executed at the same moment of time. Threads of different warps can be at different stages of the program running. This method of data processing is called SIMT (Single Instruction - Multiple Threads). Warp's management is carried out at the hardware level.

In some cases GPU is slower than CPU, but why?

Don't try to run easy tasks on your GPU. I'm gonna explain that.

Delay - it's the waiting time between requesting for a particular resource and accessing that resource;
Bandwidth - the number of operations that are performed per unit of time.

So, the main question is: why does a graphics processor sometimes stumble? Let's find it out!

We have two cars:

passenger van - speed 120 km/h, capacity of 9 people;
bus - speed 90 km/h, capacity of 30 people.

If an operation is the movement of one person at a certain distance, let it be 1 kilometer, then the delay (the time for which one person will pass 1 km) for the first car is 3600/120 = 30s, and the bandwidth is 9/30 = 0.3.
For the bus it's 3600/90 = 40s, and the bandwidth is 30/40 = 0.75.

Thus the CPU is a passenger van, the GPU is a bus: it has a big delay, but also a large bandwidth.

If the delay of each particular operation is not as important as the number of these operations per second for your task it is worth considering the use of the GPU.

Thoughts

The distinctive features of the GPU (compared to the CPU) are:

architecture, maximally aimed at increasing the speed of calculation of textures and complex graphic objects;
The peak power of the typical GPU is much higher than that of the CPU;
Thanks to the specialized pipelining architecture, the GPU is much more effectively in processing graphical information than the CPU.

In my opinion, the main disadvantage is that this technology is supported only by Nvidia GPUs.

The GPU may not always give you an acceleration while performing certain algorithms. Therefore, before using the GPU for computing, you need to think carefully if it is necessary in that case. You can use a graphics card for complex calculations: work with graphics or images, engineering calculations, etc., but do not use the GPU for simple tasks (of course, you can, but then the efficiency will be 0).

See ya! And remember:

When using a GPU it's much easier to slow down a program than that speed it up.

How neural network works? Let's figure it out

Petro Liashchynskyi — Sat, 19 Jan 2019 13:29:12 +0000

Hey, what's up 😁 In my previous article i have described how to build neural network from scratch with only JavaScript. Today, at the request of several people, i'll try to explain mathematical principle of neural networks. Bro, you finally will understand what under the hood of that monster is!

And first, i'm gonna tell you another secret: there's no magic, just only math 😵

This article is based on my previous one. If you don't read it yet, it's time to do that! I will use the same formulas and try to explain them. Let's go!

Preparation

I'm gonna solve XOR again 😅 It's not a joke, bro! There are many data science books start with solving it 😎 One more time i remind you XOR input table.

Inputs	Outputs
0 0	0
0 1	1
1 0	1
1 1	0

To demonstrate it let's use the following structure of neural network.

Here we have 2 neurons in input layer, 4 in hidden and 1 in output layer.

Weights initialization

The main goal of neural network training is adjusting the weights to minimize the output error. In most cases, the weights is initializing randomly and during neural net training these ones is adjusting by backpropagation algorithm.

So, let's initialize the weights randomly from [0, 1] range.

Graphically, it looks like this.

Forward propagation

Ok, let's compute neuron inputs. I will use only one input case to save time: 0 and 1 so the output will be 1.

The formula:

So, for the first neuron in the hidden layer:

net1_h = 0 * 0.2 + 1 * 0.6 = 0.6

/**

1..n, n = 2 (2 neurons in the input layer)

0 value of the first input element
1 value of the second input element

0.2 the weight from first input neuron to first hidden
0.6 the weight from second input neuron to first hidden

Understand, bro? 😏 

*/

For second one and others:

net2_h = 0 * 0.5 + 1 * 0.7 = 0.7
net3_h = 0 * 0.4 + 1 * 0.9 = 0.9
net4_h = 0 * 0.8 + 1 * 0.3 = 0.3

Now, we need one more thing - we need to choose activation function. I'll use sigmoid.

The formula and derivative:

f(x) = 1 / (1 + exp(-x))
deriv(x) = f(x) * (1 - f(x))

So, now we apply our activation to each of computed net:

output1_h = f(net1_h) = f(0.6) = 0.64
output2_h = f(net2_h) = f(0.7) = 0.66
output3_h = f(net3_h) = f(0.9) = 0.71
output4_h = f(net4_h) = f(0.3) = 0.57

We've got the output values for each neuron in the hidden layer. Graphically, it looks like this:

And now, when we've got output values for hidden layer neurons we can calculate the output value for the output layer.

net_o = 0.64 * 0.6 + 0.66 * 0.7 + 0.71 * 0.3 + 0.57 * 0.4 = 1.28
output_o = f(net_o) = f(1.28) = 0.78

And here we go.

Back propagation

Bro, look at the output value. What do you see? 0.78 right? If you remember the XOR table you know that we should have got 1 for this case 0 1, but we've got 0.78. That's called an error. Let's calculate that.

Output error and delta

The formula:

target = 1
error = target - output_o = 1 - 0.78 = 0.22

Now, we need to calculate the delta error. In general, that's the value by which you adjust the weights.

The formula:

You can use this site for sigmoid derivative calculation.

delta_error = deriv(output_o) * error = deriv(0.78) * 0.22 = 0.21 * 0.22 = 0.04

Hidden error and delta

Let's do the same for each neuron in the hidden layer. The formula is different a little bit.

We need to calculate the error for each neuron. Remember it, bro. Let's get started!

error1_h = delta_error * 0.6 = 0.04 * 0.6 = 0.024
error2_h = delta_error * 0.6 = 0.04 * 0.7 = 0.028
error3_h = delta_error * 0.6 = 0.04 * 0.3 = 0.012
error4_h = delta_error * 0.6 = 0.04 * 0.4 = 0.016

And again the delta!

delta_error1_h = deriv(output1_h) * error1_h = deriv(0.64) * 0.024 = 0.22 * 0.024 = 0.005
delta_error2_h = deriv(output2_h) * error2_h = deriv(0.66) * 0.028 = 0.224 * 0.028 = 0.006
delta_error3_h = deriv(output3_h) * error3_h = deriv(0.71) * 0.012 = 0.220 * 0.012 = 0.002
delta_error4_h = deriv(output4_h) * error4_h = deriv(0.57) * 0.016 = 0.23 * 0.016 = 0.003

The time has come! 😎

Now, we have all variables to update the weights. The formulas look like this.

Let's start from the hidden to the output.

learning_rate = 0.001

hidden_to_output_1 = old_weight + output1_h * delta_error * learning_rate = 0.6 + 0.64 * 0.04 * 0.001 = 0.6000256
hidden_to_output_2 = old_weight + output2_h * delta_error * learning_rate = 0.7 + 0.66 * 0.04 * 0.001 = 0.7000264
hidden_to_output_3 = old_weight + output3_h * delta_error * learning_rate = 0.3 + 0.71 * 0.04 * 0.001 = 0.3000284
hidden_to_output_4 = old_weight + output4_h * delta_error * learning_rate = 0.4 + 0.57 * 0.04 * 0.001 = 0.4000228

We've got the values too close to the old weights. It's because we chose the learning rate too small. It's a very important hyper parameter. When you choose it too small - your network will training for years 😄 Otherwise, when it's a large number - your network will train faster, but it's accuracy may be low for new data. So you have to choose it correctly. The optimal value is in range between 1e-3 and 2e-5.

Ok, let's do the same for the input to the hidden synapses.

//for the first hidden neuron
input_to_hidden_1 = old_weight + input_0 * delta_error1_h * learning_rate = 0.2 + 0 * 0.005 * 0.001 = 0.2
input_to_hidden_2 = old_weight + input_1 * delta_error1_h * learning_rate = 0.6 + 1 * 0.005 * 0.001 = 0.600005

//for the second one
input_to_hidden_3 = old_weight + input_0 * delta_error2_h * learning_rate = 0.5 + 0 * 0.006 * 0.001 = 0.5
input_to_hidden_4 = old_weight + input_1 * delta_error2_h * learning_rate = 0.7 + 1 * 0.006 * 0.001 = 0.700006

//for the third one
input_to_hidden_5 = old_weight + input_0 * delta_error3_h * learning_rate = 0.4 + 0 * 0.002 * 0.001 = 0.4
input_to_hidden_6 = old_weight + input_1 * delta_error3_h * learning_rate = 0.9 + 1 * 0.002 * 0.001 = 0.900002

//for the fourth one
input_to_hidden_7 = old_weight + input_0 * delta_error4_h * learning_rate = 0.8 + 0 * 0.003 * 0.001 = 0.8
input_to_hidden_8 = old_weight + input_1 * delta_error4_h * learning_rate = 0.3 + 1 * 0.003 * 0.001 = 0.300003

That's it! Finally 😉

Conclusions

Oh, finally we did all the math stuff! But we only did that for one training set - 0 and 1. For our problem we solve (XOR) we have 4 training sets (see the table above). That means you have to do the same calculations we just did above for each training set! Brrr, that's terrible 😑 Too much math 😆

So, in machine learning when you do one forward propagation step (from the input layer to the output) and one backward (from the output layer to the input) for one training set it's called an iteration. Another important term is epoch. Epoch counter is iterating when you pass through your neural network all the training sets. In our case, we have 4 training sets. One iteration means one training set passed through neural network. When all training sets passed through a network - here we have one epoch. Then: 4 iterations equals 1 epoch. Understand, bro? 🤗 In general, more epochs - a higher accuracy, less epochs - a lower accuracy.

That's it. No magic, only math. Hope, you've understood it, bro 😊 See ya! Happy coding 😇

Creating of neural network using JavaScript in 7 minutes!

Petro Liashchynskyi — Sat, 12 Jan 2019 12:58:53 +0000

Hey, what's up 😁 Today, i'm gonna tell you how to build a simple neural network with JavaScript by your own with no AI frameworks. Let's go!

For good understanding you need to know these things:

OOP, JS, ES6;
basic math;
basic linear algebra.

Simple theory

A neural network is a collection of neurons with synapses connected them. A neuron can be represented as a function that receive some input values and produced some output as a result.

Every single synapse has its own weight. So, the main elements of a neural net are neurons connected into layers in specific way.

Every single neural net has at least an input layer, at least one hidden and an output layer. When each neuron in each layer is connected to all neurons in the next layer then it's called multilayer perceptron (MLP). If neural net has more than one of hidden layer then it's called Deep Neural Network (DNN).

The picture represents DNN of type 6–4–3–1 means 6 neurons in the input layer, 4 in the first hidden, 3 in the second one and 1 in the output layer.

Forward propagation

A neuron can have one or more inputs that can be an outputs of other neurons.

X1 and X2 - input data;
w1, w2 - weights;
f(x1, x2) - activation function;
Y - output value.

So, we can describe all the stuff above by mathematical formula:

The formula describes neuron input value. In this formula: n - number of inputs, x - input value, w - weight, b - bias (w e won't use that feature yet, but only one thing you should know about that now - it always equals to 1).

As you can see, we need to multiply each input value by its weight and summarize products. We have sum of the products of multiplying x by w. The next step is passing the output value net through activation function. The same operation needs to be applied to each neuron in our neural net.

Finally, you know what the forward propagation is.

Backward propagation (or backpropagation or just backprop)

Backprop is one of the powerful algorithms first introduced in 1970. [Read more about how it works.]

Backprop consists of several steps you need apply to each neuron in your neural net.

First of all, you need to calculate error of the output layer of neural net.

target - true value, output - real output from neural net.

Second step is about calculating delta error value.

f' - derivative of activation function.

Calculating an error of hidden layer neurons.

synapse - weight of a neuron that's connected between hidden and output layer.

Then we calculate delta again, but now for hidden layer neurons.

output - output value of a neuron in a hidden layer.

It's time to update the weights.

lrate - learning rate.

Buddies, we just used the simplest backprop algorithm and gradient descent😯. If you wanna dive deeper then watch this video.

And that's it. We’re done with all math. Just code it!!!

Practice

So, we’ll create MLP for solving XOR problem (really, man? 😯).

From the simplest things to the hardest, bro. All in good time.

Input, Output for XOR.

We’ll use Node.js platform and math.js library (which is similar to numpy in Python). Run these commands in your terminal:

mkdir mlp && cd mlp 
npm init 
npm install babel-cli babel-preset-env mathjs

Let’s create a file called activations.js which will contain our activation functions definition. In our example we’ll use classical sigmoid function (oldschool, bro).

Then let’s create nn.js file that contains NeuralNetwork class implementation.

It seems that something is missing.. ohh, exactly! we need to add trainable ability to our network.

And just add predict method for producing result.

Finally, let’s create index.js file where all the stuff we created above will joined.

Predictions from our neural net:

Conclusions

As you can see, the error of the network is going to zero with each next epoch. But you know what? I’ll tell you a secret — it won’t reach zero, bro. That thing can take very long time to be done. It won’t happens. Never.

Finally, we see results that are very close to input data. The simplest neural net, but it works!

Source code is available on my GitHub.

Original article posted by me in my native language.