DEV Community: Boussebha Wassim

Saving the state of an interactive container

Boussebha Wassim — Fri, 27 Sep 2024 16:16:35 +0000

Sometimes we need to start a docker container in an interactive shell for testing purposes , and during the process you might wrote some application code , downloaded some big chunks of files, or configured the environment according to your use-cases , but starting the container interactively will make it destroyed once an interrupt is done ( 0 or -1 ) .

docker run -it --rm --mount type=bind,source="$(pwd)"/work,target=/work -p 8888:8888 opencvcourses/opencv:440

-it starts an interactive shell. This switch is always needed to start the container. Otherwise, it will start and stop instantly.
--rm specifies to kill the container after it is exited.
--mount creates persistent storage to save all the work. Read more about it here⁠.
-p is used to expose a container's port to the host.
Note: Run this command in the parent directory of work folder

you can save the state by commiting the changes to the pulled image.
$ docker ps

$ docker commit <container_id> repo/testimage:version3

While this will work, this approach is considered poor practice by the Docker community. Best way to go is using a Dockerfile.

Yeah, I think that's the right way for production for sure. But many of us are just using docker locally as a slimmed down version of virtualbox, and just want the damn state saved, exactly as it is. And don't want extra volumes on our local drives either

Install OpenCV 4.5 on Ubuntu 22.04

Boussebha Wassim — Fri, 27 Sep 2024 05:50:45 +0000

OpenCV (Open Source Computer Vision Library) is a library of programming functions mainly for real-time computer vision.

But it's installation can be very tricky in an environment like Linux, so let's follow a correct installation process

Build OpenCV for Python and C++ from sources

Let's open the most recent release of opencv to the date of this video capturing:
https://github.com/opencv/opencv/releases/tag/4.5.1
Create a tmp folder for all archives:
mkdir ~/opencv4.5-tmp && cd ~/opencv4.5-tmp
We need to download opencv sources:
wget https://github.com/opencv/opencv/archive/4.5.1.zip -O opencv.zip
We need to download opencv-contrib sources:
wget https://github.com/opencv/opencv_contrib/archive/4.5.1.zip -O opencv_contrib.zip
Unzip the opencv files:
unzip opencv.zip
Unzip the opencv-contrib files:
unzip opencv_contrib.zip
Move the files to simple directories:
mv opencv-4.5.1/ opencv
Move opencv-contrib files to simple directories:
mv opencv_contrib-4.5.1/ opencv_contrib
Make build directory:
cd opencv && mkdir build && cd build
Copy and run the following command. Install cmake if it is not available on the system.

   cmake -D CMAKE_BUILD_TYPE=DEBUG \
         -D CMAKE_INSTALL_PREFIX=~/opencv4.5-custom \
         -D OPENCV_EXTRA_MODULES_PATH=~/opencv4.5-tmp/opencv_contrib/modules \
         -D OPENCV_GENERATE_PKGCONFIG=ON -DOPENCV_GENERATE_PKGCONFIG=ON \
         -D BUILD_EXAMPLES=ON ..

Make the project:

   make -j4

Install opencv:

   sudo make install

Ensure that it is updated in the library storage:

   sudo ldconfig

## Configure C++ project to work with opencv

Open your editor of choice (vim in my case)
Create a folder "~/projects/HelloOpenCV"
Paste the main.cpp code to the main.cpp
Firstly, let's try to compile our application with g++ as usual:

   g++ -Wall -o main main.cpp

We see that it cannot find our library headers

For that we need to provide path to the headers and linker flags. The best way to find them all in one place is to use the utility module pkg-config. Remember we provided an additional argument to our cmake generation? So, let's execute the following:

   export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/home/parallels/opencv4.5-custom/lib/pkgconfig
   pkg-config --cflags --libs opencv4

Add all flags to compilation command:

   g++ -Wall -o main main.cpp $(pkg-config --cflags --libs opencv4)

or a more explicit version for our particular sample:

   g++ -Wall -o main main.cpp \
      -I/home/parallels/opencv4.5-custom/include/opencv4 \
      -L/home/parallels/opencv4.5-custom/lib \
      -lopencv_highgui -lopencv_videoio -lopencv_imgcodecs -lopencv_core

Real-time Disk Size Expanding in a Linux virtual machine

Boussebha Wassim — Fri, 05 Jul 2024 15:14:35 +0000

I was compiling the Linux kernel from source and i had a vm running AlmaLinux with 20GB Logical volume, so i attained the maximum size, i had to expend my disk size without losing my data, the progress of a 12 hours compilation process(yes heavy kernel, and a weak hardware)

this solution is used in storage management in cloud services providers AWS, Azure ..

Linux Distribution : AlmaLinux 9.4 on Vmware Workstation 7

When you expand the disk size of your Linux virtual machine (VM) on VMware, you must adjust the partitions within the system to utilize the additional space.

This guide will walk you through the steps required to resize the partitions and filesystems using bash.

Prerequisites
Before you begin, ensure you have:

Expanded the disk size of your VM in VMware.

Root or sudo access to your Linux VM.

Step 1: Identify the New Disk Size
First, verify the current disk and partition sizes using the lsblk command:

lsblk

The output will show all block devices and their partitions. Identify the disk you have expanded. For example:

NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sr0                 11:0    1  988M  0 rom  
nvme0n1            259:0    0  100G  0 disk 
├─nvme0n1p1        259:1    0    1G  0 part /boot
└─nvme0n1p2        259:2    0   19G  0 part 
  ├─almalinux-root 253:0    0   17G  0 lvm  /
  └─almalinux-swap 253:1    0    2G  0 lvm  [SWAP]

Step 2: Resize the Partition
Use the growpart utility to resize the partition. If it's not already installed, you can install it with:

sudo dnf install cloud-utils-growpart

Then, resize the partition nvme0n1p2:

sudo growpart /dev/nvme0n1 2

Step 3: Resize the Physical Volume (PV)
Next, resize the physical volume to recognize the expanded partition:

sudo pvresize /dev/nvme0n1p2

Step 4: Verify the Physical Volume Size
Verify the changes using the pvs command:

sudo pvs

Step 5: Resize the Logical Volume (LV)

Assuming you want to allocate all the new space to the root logical volume (almalinux-root):

sudo lvextend -l +100%FREE /dev/almalinux/root

Step 6: Determine the Filesystem Type
Before resizing the filesystem, determine the filesystem type. You can do this with the df -Th or lsblk -f commands:

df -Th

lsblk -f

Step 7: Resize the Filesystem
Depending on the filesystem type, use the appropriate command to resize the filesystem.

For ext4:

sudo resize2fs /dev/almalinux/root

For xfs:

sudo xfs_growfs /

Step 8: Verify the Changes
Finally, verify that the new space is available using the lsblk and df -h commands:

lsblk
df -h

Conclusion

By following these steps, you can effectively utilize the additional disk space after expanding your Linux VM disk on VMware.

CAFE BABE, if you know , you know JVM enthusiast.

Boussebha Wassim — Sat, 30 Dec 2023 22:04:20 +0000

CAFE BABE, if you know , you know JVM enthusiast.

Each program has a magic word in it's machine code representation that defines it's identity among multiple other files .

if you open a compiled java file aka file.class with a hex editor like xxd tool in Linux , you'll find the first hex words : CAFE BABE .

But why it is so magical that it has to add in each class file is probably the most important question to be asked.
Well, James Gosling explained why: –

We used to go to lunch at a place called St Michael’s Alley. According to local legend, in the deep dark past, the Grateful Dead used to perform there before they made it big. It was a pretty funky place that was definitely a Grateful Dead Kinda Place. When Jerry died, they even put up a little Buddhist-esque shrine. When we used to go there, we referred to the place as Cafe Dead. Somewhere along the line, it was noticed that this was a HEX number. I was re-vamping some file format code and needed a couple of magic numbers: one for the persistent object file, and one for classes. I used CAFEDEAD for the object file format, and in grepping for 4 character hex words that fit after “CAFE” (it seemed to be a good theme) I hit on BABE and decided to use it. At that time, it didn’t seem terribly important or destined to go anywhere but the trash can of history. So CAFEBABE became the class file format, and CAFEDEAD was the persistent object format. But the persistent object facility went away, and along with it went the use of CAFEDEAD – it was eventually replaced by RMI.

How your program's functions are handled in memory ?

Boussebha Wassim — Fri, 29 Dec 2023 15:40:54 +0000

Because i always forget how the kernel is handling the program's functions in the call stack , i'll write about it so i come back after months or someone else will :) .

So hello , let's do it in C in a x86-64bits machine, why ? because it's my favorite language .

first , when you compile your C code ( gcc -c main main.c ), you get an object file main.o that will be linked later ( manually or dynamically ) with other already compiled standard code ( like your famous printf() ) , you get an executable file , ELF64 in my case.
okay , when you disassemble either with objdump or gdb , you can see the assembly code of your program :
gdb ./main
disassemble function

you'll find some sections like .text , .bss or .data , heap and the stack these are the sections of the virtual address space of your program .

what concern us is the .text section , it contains the code ( instructions to be executed [ for example int x = 3 will have the equivalent of movl $0x3,-0x4(%rbp)])

okay , good , what concerns us is how the functions main() and function() are handled in the stack (the section where the functions resides) .

before , the CPU unit has some special registers called %rbp , %rsp ( the base pointer and the stack pointer ) to manipulate the scope of the functions.

%rbp : is used to access the memory words by adding values to it like the example we had : movl $0x3,-0x4(%rbp)
%rsp : will always points to the top of the stack.

so the first thing we need to do is to push the value of the old %rbp ( because it's used by other functions to do the same thing , in this case the main() function which called the function() ) .

then , we push the return address to the stack so the main() function resumes it's execution where it stopped ( in this case it will return to line 13). , this information is retrieved from register r14 in ARM processors for example.

then we'll put the new value of the base pointer %rbp ( the one we said used for accessing memory words in the scope ) to the value of %rsp , so the top of the stack .

this process of handling the %rbp and %rsp is called function prologue and it's done in each function start ( as you see in the assembly code ) .

this is how it's done with the initialization of the function scope in multiple architectures and systems similarly.

now we'll use the function call stack by putting the local variables , the function parameters etc , these specific details follows calling conventions:
https://lnkd.in/eUNNAiXH

one of the conventions to follow in x64 architectures for example is SYSTEM V
the registers used for function parameters handling
The first is placed in rdi, the second in rsi, the third in rdx, and then rcx, r8 and r9. Only the 7th argument and onwards are passed on the stack , the left most parameter is passed first on the stack , and then the old value of the base pointer , then it's followed by the local variables of the function.

what the heck is locality of reference , why i will waste brain cells bytes to know it?

Boussebha Wassim — Mon, 25 Sep 2023 23:51:05 +0000

Locality of Reference is a critical Computer Science topic and helps improving the software's performance if the programmer is aware of it .

Programs that tend to exhibit good locality are those who're constituted of components that tend to reference data items that are near other recently referenced data items or that were recently referenced themselves .

There is 2 types of locality :

1- Temporal locality : if a main memory's word is referenced once , and there are potential references in the near future , it should be in the cache memory.

2- Spatial locality : if a memory location is referenced once, then the program is likely to reference a nearby memory location in the near future ; A good example is a programming block which is logically distinguishable that is : Traversing a contiguous array of data using a loop statement .

Okay thank you for giving me information that i can get from Wikipedia , how the heck that will help me improve my programming skills & performance ?

Answer : We have the following example that performs matrix values sum

the sumarraycols() function in Figure 6.19(a) computes the same result as the sumarrayrows function in Figure 6.18(a). The only difference is that we have interchanged the i and j loops.

But What impact does interchanging the loops have on its locality?

The sumarraycols() function suffers from poor spatial locality because it scans

the array column-wise instead of row-wise. Since C arrays are laid out in memory row-wise.

The examples retrieved from the book : Computer Systems - Programmer's perspective.

How the heck a C program is compiled from human comprehensible language to 0's and 1's

Boussebha Wassim — Tue, 28 Feb 2023 01:38:09 +0000

You may wondered how the programming languages lexemes are turned into a series of 0's and 1's aka binary files ?

Figure 2.11 - Book reference : Operating system concepts.

Let's discover that together step by step :

Usually programs a.out or program.exe ( games , word processors , web browsers ..) resides in your disk as a binary executable file , and in order to execute one of them , you need to fetch it from disk and put in memory , and it waits until it's scheduled to run and becomes a running program also known as a "process" which have a specific amount of memory addresses and executed by the units of cpu's core with the help of specific registers like program counters , IR ... and general-purpose registers .

but we don't write operating systems and Enterprise systems with binary right?

We need a human readable language , let's chose a compiled programming language like C to know the process of compilation .

After writing your program.c source code with human readable C , the source code is compiled & assembled to a relocatable object file program.o ( machine code that can be disassembled and turned into a geek-readable assembly ) ( we will talk about phases of compilation ( lexing , parsing , AST...) in future articles ) .

So we have now a object file , but we may called several functions like printf() which is also written by system programmers before to give us a layer of abstraction and increase our productivity , and compiled aka turned into an object file , it's part of GCC .

we need now to link all those object files and make them a single binary executable file that can be loaded into memory , this mechanism is done by what we call a " Linker " . (this is static linking)

the mechanism of loading the binary executable file into memory so it can be executed by one of the cpu's core is done by the loader , it's also responsible of addresses reallocation in memory..

You may heard about DLL ( dynamically linked libraries ) files in Windows right ? now we speak about dynamic linking . actually , most systems allows programs to dynamically link the libraries even if the program is already loaded and started execution ( that's good from memory optimization prospective, because the programmer may include a library and doesn't use it during runtime , i am sure we all did that before )

you know that modern systems allows multiple processes to share DLL ? yes they do!

And now you have a binary executable program into memory , also known as a process , with it's own memory addresses .

I will talk deeper next time about the compilation phases , and how the cpu fetch & decode & execute instructions & data from the bounded virtual memory of the process.

Why the heck the _state variable in task_struct structure is 4 bytes

Boussebha Wassim — Tue, 28 Feb 2023 01:34:32 +0000

According to Robert Love the Author of the book Linux Kernel developement & Linux kernel source :

In the Linux kernel , specifically in the PCB ( process control block ) which a structure that represents the process in the kernel .

The process state is represented by a combination of six state flags, which are:

TASK_RUNNING: The process is currently running or ready to run.

TASK_INTERRUPTIBLE: The process is waiting for a specific event to occur and can be interrupted by a signal.

TASK_UNINTERRUPTIBLE: The process is waiting for a specific event to occur and cannot be interrupted by a signal.

__TASK_STOPPED: The process has been stopped (e.g. by a SIGSTOP signal) and can be resumed later.

__TASK_TRACED: The process is being traced by another process (e.g. a debugger).

TASK_DEAD: The process has terminated and is waiting to be reaped by its parent process.

Each of these flags can be set or cleared to represent the different states that a process can be in. For example, a process that is both TASK_INTERRUPTIBLE and TASK_UNINTERRUPTIBLE might be waiting for a disk I/O operation to complete, and can be interrupted by a signal but cannot be killed until the I/O operation is finished.

In other words, a process can have multiple states at once, depending on which combination of flags is set.

This state is saved in an *UNSIGNED INTEGER *, that means 32 bits , that means 2³² possible values , but we need only 2⁶ , so 64 possible states as a maximum .

the reason is the following :

There are some predefined functions that manages the process's state such as READ_ONCE/WRITE_ONCE that obliges this variable to be an unsigned INT , it's used to be a volatile Long ; here are the commits :

https://github.com/torvalds/linux/commit/2f064a59a11ff9bc22e52e9678bc601404c7cb34#diff-f8d8a1568ae83bbff6f40f9c70559a4f7dbf426a397131ba9d4fbfb947ea5222R669

Thank you for reading <3 .

Why the heck gateways are invented in computer networks ?

Boussebha Wassim — Tue, 28 Feb 2023 01:24:34 +0000

I've always wondered why the default gateway is needed in computer networks , I realized finally, so i wanted to share that with you guys :) .

let's take an example , an HTTPS request to Google.com , a DATA chunk is generated , which contains data like the destination google's IP address resolved from DNS , type of HTTP request ( GET OR POST ..) etc...
The encapsulation process will start in the TCP/IP Suite installed in your device , and it's like the following :

each layer will perform some work on this DATA sent to Google's Server , the first inferior level is the Transport layer , SSL/HTTP mainly runs on top of TCP , so the connection oriented protocol will prepare the TCP tunnel by performing the TCP handshake with the server and allow TCP tunneling to exchange both data and metadata .

The TCP will also segment the DATA and does the sequencing to avoid incoherent DATA in the destination , also adds both source and destination operating system processes ports( TCP Does a lot of things , google to know more ).

So we will have now a bunch of segmented , sequenced pdu's called Segments , with DATA and the TCP Header , it passes to the next layer .

The Internet Layer , for example IPv4 , the layer will add the IPv4 header that contains importantly the source and destination IP address, so we will have now a packet , an important header will be added , it's the TTL , time to live , how much time the packet will live in the Internet routers around the world , each time it passed a Router , the value of the TTL will get decremented until it reaches the zero and the packet get destroyed .

Now the most important thing , we have an IPv4 packet ready to be routed in the Internet , but it needs to pass through Data Link layer , the Ethernet ( even 802.3 or 802.11 ) , this layer will add to the packet both source and destination MAC address to make sure that the data is sent to the correct device , since the MAC address is the unique network identifier for each device, and ip addresses can change .

It's possible for the device that sent this HTTPS request to know the destination mac address if we're on the same local area network , that's thanks to the Address resolution protocol by sending a broadcast ipv4 packet to all devices in the same network and saving results on the ARP table which contains the mac address for each device connected to this specific router interface.

the ethernet frame need to be transformed into bits and sent via UTP or fiber optic cables to our router so it routes it to the google's server .

In this situation , the device needs to know the mac address of the router , so the ethernet frame will be sent to the router , and routers communicates with each other with layer 3 addresses aka ip addresses .
so the device will just need to replace the destination address with the gateway aka the local ip address of the router , and get it's mac address via the ARP table , and put it in the ethernet frame .
and the packet is sent.