DEV Community: Marcell Cruz

Web Essentials PART 2

Marcell Cruz — Fri, 28 Feb 2025 16:18:42 +0000

In the last part 1 I talked about things that are good to practice and remember besides the usual interview stuff like data structures, algorithms, and system design, in this part, I'm expanding that adding material to practice remembering those concepts simply and effectively.

Space Repetition With Anki Cards

Download Anki Cards anki Anki card is fantastic it works natively on any operating system and Android and it's free, after downloading Anki cards you can download the cards that I created here

Download that repo, click on code and then download zip.

unzip the file inside there's a bunch of .apkg files, you can import these files to anki.

open anki click file > import, choose the .apkg files, click import and that should be it.

Now you have an excellent tool to memorize some of the basics, you should keep all cards in your deck grey, which means that you're up to date with your practice you can read more about anki cards here https://leananki.com/how-to-use-anki-tutorial/ it looks too simple to be true but it's very effective

Web Essentials

Marcell Cruz — Thu, 30 Jan 2025 01:12:11 +0000

TL;DR
if you're just interested in the list, scroll down

I was pairing with a new developer recently and he was stuck trying to fix an issue in a react form, and I noticed that the reason why he couldn't fix the issue was a lack of understanding of how a form works under the hood, if he knew of the top of his head the very basics of how a html form works he would be able to fix the issue very easily. what he needed to understand wasn't complicated but he needed to understand that in order to proceed, searching on google or asking chatgpt didn't help him, because he didn't knew what to search, this experience got stuck in my head and I begin wondering about other things like this, small things that on it's own doesn't mean much, but if you put them together you start seeing the difference between moving faster and enjoying what you're doing or moving slowly and being bored, with time you learn this small important things and you don't even think about them anymore, you just solve the issue and move on, you don't realize that if you didn't know about that small thing you would be stuck for hours in this problem and that you needed to know that in order to fix the issue that you're working one, this post is a attempt to list seaminly small things like these that I think are important to make web development easier and more enjoyable, I believe memorizing these things is important, because you don't spend any efforce thinking about them, which makes programming more enjoing furthermore these things unlock intuition on how other related things work making you know how to solve those other issues or what to search or ask an AI, if you want to memorize these things I would suggest something like chunking, this list is not a list of things that you must memorize to be a good engineer, and don't feel bad if you don't know any of them, I didn't, until I realize what I just said and made a effort of memorizing them.

Why does Memorization matter?

you might ask why memorization matters if I can ask ChatGPT, well there's the thing that I mentioned before and also chunking, and personally more importantly than any of that is as I said many times in this post enjoyment it's very hard to enjoy something that you need to stop constantly to search or ask an AI, it's just boring and frustrating.

The List

The challenge here is, can you write these of the top of your head?
can you open your editor and write a small script that implements the solution right now? without googling, autocomplete or AI.

This is not a complete list by any means please let me know what you would add to the list.

High Level Programming Language(Ruby, Javascript, Python, Elixir)

Any high-level programming language will do!

list CRUD(create read update and delete)
hash CRUD
traverse a data structure recursively
Create a class(or the basic structure of your language)
Create a function/method
write a test
how to declare import and use a module
mock a method
mock a class
http server
connect to a database
GET, POST, PATCH and DELETE requests
File IO,
- create
- close
- append
- delete
- check the existence of a file
- loop through files in a folder

SQL

connect to a database
create a database
create a table
insert into a table
delete from a table
like, in, not null
join

HTML Forms

this implies that you know how to read the result in your server as well

post, patch and delete methods
send input text
send a file
send selected option in a select

HTML/CSS

I feel like you don't need to memorize any of this stuff, learn how it works, and then ask an AI and you should be fine, please let me know if I'm wrong.

Frontend and backend frameworks

please see comment at the end

Linux Basics

Create a user
check file permissions
give/remove file permissions
Create a group
add a user to a group
add an executable to the path
create an executable
shebang
check running processes
kill a process
Check the program running in a specific port
run something in the background
check if a service is running
enable/disable a service
start/stop a service

Docker

create an image from a docker file
pull image
list downloaded images
remove image
run image
list containers
start/stop containers
remove container
run a shell inside the container
remove stopped containers
remove unused images

nginx

reverse proxy to a server
very basic load balancing
very basic caching

I could add many more things and probably will as I realize other stuff that might be important to memorize, I didn't add any framework-specific stuff because I think this varies too much depending on the framework that you're using but I have a list for the frameworks I use as well.

Thanks for reading.

[Boost]

Marcell Cruz — Thu, 30 Jan 2025 00:23:32 +0000

"ngrok" from scratch

Sibelius Seraphini for Woovi ・ Jan 28

#ngrok #ssh #nginx

Basic methods for comprehension

Marcell Cruz — Thu, 16 Nov 2023 01:29:46 +0000

tl;dr

Atomistic Approach

Break things down into the smallest reasonable units
Ask: What does this thing do? What are the relationships with other units?

Holistic Approach

Break things down into the biggest reasonable units
Ask: What does this unit do? What are the relationships with other units?

Combination of Both Approaches is Usually the Best

Atomistic

These are the two ways of thinking about something:

In the Atomistic Approach, you break things down into smaller components, trying to explain them and understand their relationships with other components. This approach is widely understood and commonly used. It involves breaking things down into the smallest reasonable units. The term "reasonable" is crucial here, signifying a unit that is small but still meaningful for the intended purpose. For example, thinking about a function as a unit makes more sense than breaking it down into each line of code. Questions about the unit arise:

What does the unit do?
What are the relationships with other units?

Holistic

The Holistic Approach is the opposite. It involves breaking things down into the biggest units and then relating these big units to answer the same questions. For example, consider what an entire module/file/class does and how it relates to others. You could even think about what an entire app does and how it relates to other apps.

Combination of Both Approaches is Usually the Best

Typically, a combination of both Atomistic and Holistic approaches yields the best results. Start with a Holistic view, then progressively break down the units into smaller components.

How To Create An UI Menu In Neovim

Marcell Cruz — Fri, 02 Jun 2023 02:26:44 +0000

One of the great things about neovim is the ability to easilty create plugins using lua, lua is a very simple language and the way that it's integrated with neovim is really straight foward, when neovim is initiated it reads a file called init.lua in one of the paths configured, you can check the path by opening neovim and typing :h rtp for run time path you should see something like this

the first path is the one that matters for us, it means that neovim will search for the file inside ~/nvim we can then go ahead and create a file there or use the file that you already have to create a function, just to test things out.

we can create a function to open a pop up menu using plenary.popup, you need to install neovim plenary if you don't already have it here's the link https://github.com/nvim-lua/plenary.nvim, here's the function to show a popup menu using neovim plenary



local popup = require("plenary.popup")

local Win_id

function ShowMenu(opts, cb)
  local height = 20
  local width = 30
  local borderchars = { "─", "│", "─", "│", "╭", "╮", "╯", "╰" }

  Win_id = popup.create(opts, {
        title = "MyProjects",
        highlight = "MyProjectWindow",
        line = math.floor(((vim.o.lines - height) / 2) - 1),
        col = math.floor((vim.o.columns - width) / 2),
        minwidth = width,
        minheight = height,
        borderchars = borderchars,
        callback = cb,
  })
  local bufnr = vim.api.nvim_win_get_buf(Win_id)
  vim.api.nvim_buf_set_keymap(bufnr, "n", "q", "<cmd>lua CloseMenu()<CR>", { silent=false })
end

The code is pretty self explanatory the only part that is a little confusing is the cb parameter, this parameter is the function that you gonna call when someone intereact with the popup menu, also we create a keymap for the popup buffer to close the popup when you hit q



vim.api.nvim_buf_set_keymap(bufnr, "n", "q", "<cmd>lua CloseMenu()<CR>", { silent=false })

This part can also be a little confusing.
and the following is the code to close the UI



function CloseMenu()
  vim.api.nvim_win_close(Win_id, true)
end

this is the function that we gonna call when we hit q when the pop up menu is openened

Now let's create another function to call the function that shows the menu



function MyMenu() 
  local opts = {}
  local cb = function(_, sel)
    print("it works")
  end
  ShowMenu(opts, cb)
end

We not gonna do anything here for now just open the menu with nothing inside it and then you can close the menu by hitting q

After writing this data to ~/nvim/init.lua you can load the file by running :so then you can run :lua MyMenu() to call the function and show the menu, you should see something like this

there's cool but not useful, how do we add data inside the menu? we can add data there the same way that we do in any buffer, just by setting lines of text, the plugin that we gonna build is just a list of projects that we can chose from and the cd in to the project from vim.

we can add the lines of text by filling out opts the variable that we pass to the menu like this



function MyMenu()
  local opts = {
    "/home/me/myproject1",
    "/home/me/myproject2",
    "/home/me/myproject3",
  }
........

this is just a list of things to show in the menu if you do that you should be able to see a list of those things now

Awesome!

We're almost there, now we just need to do something when someone choses one of the thing in the list, we can do that in the cb that we pass to the function.



local cb = function(_, sel)
  vim.cmd("cd " .. sel)
end

if you hit enter on top of one of the lines, that line will be passed to this function and then we'll change neovim current directory to that path, in our example the paths aren't real but you get the idea, you just need to put a real path there, and that should do it, one upgrade that we can do is to load a file with paths instead of hardcoding the paths in the function, that way you don't need to open this lua file everytime you want to add a project to your list of projects, we can do that just by reading a simple file with a list of projects, same a list of project in ~/projects



/home/me/myproject1
/home/me/myproject2
/home/me/myproject3

now we can read this file and add to the options like this



local file = io.open("~/projects", "r") -- Open the file in read mode
local opts = {}
if file then
  for line in file:lines() do
    table.insert(opts, line)
  end
end
--- pass the opts to the file later

the last thing we need to do is map this function to a shortcut that way we can hit the short cut and show the menu where we can chose our project



vim.keymap.set("n", "<leader>o", '<cmd>lua MyMenu()<CR>')

and that's pretty much it, after you have a way of creating a menu you can thing about a lot of interesint things todo the idea presented here is just to show a easy way of creating a menu in neovim, you can use lua to do anything that you want

How To Write A Programming Language

Marcell Cruz — Fri, 03 Feb 2023 00:13:58 +0000

The Grammar
Composing Grammar Rules
The $$ Operator And Empty Grammar Rules
Rule Types
Parsing The Language
AST

Recently I wrote a very simple programming language, here is the repo, the language is extremely small, my objective was to write a programming language that can do fibonacci, that's it.
fib written in square

[>:fib n:
  [:if n < 3:
    return n - 1
  ]
  return [:fib n - 1] + [:fib n - 2]
]
[:fib 10]

[:print n]

it works!!!

It uses one of the most popular tools to write the grammar, the AST is very minimal and is executed in C, today I'm gonna explain the main components of a programming language trying to be as concise as possible.

The Grammar

This is the heart of the language, This is how you interact with the language, the grammar rules define how to write code for the language, I used flex and bison to write the grammar, the grammar of the language is written on square.y skim through those rules just to get a idea of how they work.

Here's a grammar rule from the language

ID EQ NUM

and it defines how to interpret assignments

n = 2

You can think about the grammar rules as a way of organizing regex blocks, each token (ID, EQ, NUM) is a regex block

identifier  ([a-z]+)  // this is ID 
equal       ([=])     // this is EQ
numbers     ([0-9])+  // this is NUM

You organize the blocks in a way that you want the blocks to be matched in the language, so when the parser sees

x = 99

it's going to first transform that in the tokens ID EQ NUM then search for a rule that matches that, after finding the rule, the parser is going to execute the body of the rule

ID EQ NUM {
  struct scope * number = (struct scope *) 
  malloc(sizeof(struct scope));
  (number)->type = "number";
  (number)->value = $3;

Think about the body of the rule as a function that is going to be executed when the parser finds that match, you can access the value in the tokens using $1, $2, $3, here is where you build your AST, more on that later

Composing Grammar Rules

You can compose grammar rules, for example, in the language we have function calls that work like this [:add n] but you can also call the function passing the value directly instead of a variable [:add 10] instead of creating two different grammar rules for each case, we create a grammar rule with these two options and use it in the function call grammar rule.

OPBRA IDFUNC fcallparam CLBRA {

fcallparam: ID {
              ..body 
          } 
          | NUM {
              ..body
          }
;

OPBRA is [ IDFUNC is the name of the function :add in this case, fcallparam can be a ID identifier, basically a variable, or NUM, NUM is a number, you're free to compose the rules how many times you want.

The $$ Operator And Empty Grammar Rules

The $$ operator can be used to save the result of the composing grammar rule to be used in another grammar rule, it's easier to see the utility in action, in the last example
we can save anything on $$ than use it in the composing grammar rule

fcallparam: ID {
  $$ = "Hello There!!" 
}

OPBRA IDFUNC fcallparam CLBRA {
  printf($3) // $3 here is "Hello There!!"  
}

remember that we can access the tokens with $1, $2, $3 etc...
So now $3 is "Hello There!!".

If You just want to use the token directly you don't need the body, example

fcallparam: ID | NUM

OPBRA IDFUNC fcallparam CLBRA {
  // $3 will be either a identifier or a number depending on 
  // the match   
}

the default behavior of any rule without a body is

$$ = $1

Rule Types

A rule also has a type, to return a string like we did in the fcallparam example, the rule fcallparam needs to be a string, we define this in the beginning of the parser when we connect the parser with the lexer, you can see all definitions here

In the toy language fcallparam is of type node which is a node of the AST
%type <node> fcallparam

Parsing The Language

With the information that we have above we can begin to understand how to parse the language, and how to create the AST, for example we could create a print like function called output in the language.

In the source file…

  output("hello world")

In the parser...
We parse the contents with the right rules

OUTPUT OPPAR STR CLPAR {
  // here we just print the content inside the parenteshis
  printf($3)
}

for very simple rules we could do something like that and execute the code that we created, the problem is that programming languages have concepts like functions that postpone code execution, you cannot execute what you see directly because we have definitions that are going to be executed when the program tells you to, for example if output was inside another function we can not execute it right away, this is one of the reasons why we need a AST, the AST is a data structure that represents the code but can be executed easily, we don't need to parse anything, just traverse the data structure and execute it.

function test() {
  output("hello world")
}

in this case we cannot execute output as soon as we identify it in the parser, we need to wait for a function call that calls test(), then we execute what's inside test's body, to solve this problem we need to breakdown the parsing and the executing in to different things, that's where the AST comes in.

AST

Abstract Syntax Three is just a data structure like an linked list or binary three, you can use nested array as an AST, and some languages do use it, but we're going to use a linked list instead, this is the data structure that I used in the language

struct scope {
  char * type;
  char * extra;
  struct arg * args;
  int value;
  struct scope * scopes;
  struct scope * next; 
  struct scope * prev; 
  int return_value;
};

This is essentially a doubly linked list after I finished the language I realized that it could have been much easier but I didn't have the patience to change everything so that's what we have right now :P, it's easier to understand what it represents with an example

{
  .type = "function",
  .extra = "global",
  .scopes = {
    .type = "body",
    ...
    .scopes = {
      .type = "function",
      .extra = "fib",
      .args = {
        .key = "n", 
        .value = 0,
      }
      .scopes = {
        .type = "body",
        .scopes = {
          .type = "if"
          ...
        }
      }
    }
  }
}

The root node is a function called global, same thing as main in other languages like C, this node has a body and the body
has another node function called fib which is the declaration of the fib function, this node has a if node and so on..., I hope you get the idea, the AST is just another way of representing the code, this is what we called a IR or Intermediate Representation the reason why we do this is to make it easier to do something with that data, specially traverse it, it's very hard to traverse source code, so instead of doing that we transform the source code in to a different data structure and work with that instead
so this

[:fib n
  [:if n < 3
  ]
]

becomes this

{
  .type = "function",
  .extra = "fib",
  .scopes = {
    .type = "body",
    ...
    .scopes = {
      .type = "if",
      ...

The next thing that we need to do is execute the AST, a lot of popular languages like Python, Ruby and Java have an additional IR representing the code as bytecode and executing the bytecode in a virtual machine for optimization, but since this is a toy language we just execute the AST itself

Executing The AST

Executing the AST is pretty straighforward we pass the AST to an exec function and this function traverse and execute the AST following the rules of the language, this is the file that creates the AST and calls exec passing the AST square.y , the whole thing is less then 400 lines, and it could be much simpler as I said before, for example, the languages accepts evaluating statement in returns, so you can do things like this

return [:fib n - 1] + [:fib n - 2]

but that's not strictly necessary, you can check what the exec function does here but essentially we just traverse the AST check the node type like function if function calls etc.. and do what needs to be done, the most challenging part after building the AST is implementing scope but that can be done with pointers, each node holds the value of the variables used inside it and we can use .next and .prev to jump between scopes, I'm not gonna explain how I execute the AST here because you can simply follow the code here, it's very simple, and I think I already cover the meat of building a language, which is the parser and the AST, after you have the AST it's just a matter of executing it which is an entire different thing outside the scope of showing how a language can be implemented, if you want to build something specific I think the toy language that I've build is a good start, since a lot of the work is the actual set up.

Assembly For JavaScript Developers

Marcell Cruz — Thu, 12 Jan 2023 20:56:31 +0000

Why Should You Care?
Lines Of Code vs Instructions
Assembly Is Not A Language?
Differences Between Hardware
Why am I writing This?
Fibonacci Code
Naming the concepts
Something Like A Function
Passing Data To The Function
Creating Scope
The Stack, Another Way Of Saving Data.
Using The Stack To Save Data And Create Scope
Returning Data To The Caller
If Statements In Assembly
Conditional Logic
Recursion
Putting Everything Together
Final Code
IO In Assembly
Final Code With IO
How To Debug Assembly
Code

If you only code in high level languages and have no idea what your computer does with your code, this post is for you, no previous knowledge of assembly is required just basic knowledge of any popular language, if you can read and understand the following JavaScript code, you're good to go.

function fib(n) {
  if(n < 3) {
    return n - 1;
  }
  return fib(n - 1) + fib(n - 2);
}

fib(1); // return 0
fib(2); // returns 1
fib(3); // also returns 1 (0 + 1)
fib(4); // returns 2 (1 + 1) and so on...

We're gonna implement this in assembly breaking down each JavaScript concept and implementing the same concept in assembly.

The main problem that I've had learning assembly is that is boring, and the reason why is boring is because you have to understand a lot to do a little, and the thing that you end up doing is not that interesting, so I decided to write this post that tries a different approach.

Why Should You Care?

If you're a JavaScript developer you not going to use assembly in your daily routine and knowing assembly is not gonna make you a better JavaScript developer in any meaningful way, so why should you care?
Well, Knowing assembly will increase your understanding of computing and make you more interested in it, it'll open doors in other areas that might grab your interest, and the most important part, it'll build your narrative about programming, you need to be able to tell a history with the knowledge that you have, to be interested in it, learning assembly will increase your context appreciation and understanding of the whole field, if you feel like you're repeating yourself learning the same concepts over and over just with different names said by different people, maybe it's time to learn something new to expand your view and reignite your interest in programming, at the end of the day you can't be good at something that you don't enjoy and have no interested in, so allow yourself the opportunity of sparkling your interested once more, if you don't know C you can't fully appretiate javascript, if you don't know assembly you can't fully appretiate C, it's important to know where the things that you use come from to understand why they exist, to build meaning, Knowing how to do something is not the same as understanding it, you need a lot of context to understand something, you need to connect the dots, the more context you have the easier it's for you to understand other things in the same area and the deeper is your understanding, you can explain the "why", why something was built the way it was and "why" it's not a good idea to do it differently, what are the downside of a solution, you level of enjoyment and interest in the field also grows, because your mind have a lot more room to wonder in that particular field, so let's get to it.

Lines Of Code vs Instructions

Your computer can't execute the code that you write, let x = 99 your computer has no idea what the previous line of code means, and what do to with it, this line of code needs to be translated to instruction that your CPU can actually execute, the code that you write is called subject code or source code this code needs to be translated to object code real instructions that your CPU can execute, one line of code in a high level language gets translated to multiple instructions that a CPU can actually execute. computers are super dumb, they do not understand any of these high level concepts, they can only do very simple things like:

Move values to different places in memory
Add and subtract numbers
Jump to different instructions

And that's pretty much it, the hardware on your computer is very limited. don't get me wrong, it's amazing that we have circuitry that can actually do these things, but there's a chasm between the concepts in a high level language and what the hardware can actually do. it's the job of the person designing the language to bridge that chasm and implement these high level concepts like functions, recursion and scope using only these very simple tools and that's what we gonna do in this post, we gonna reverse engineer a solution in a high level language(JavaScript) to a low level language(assembly)

Assembly Is Not A Language?

Assembly is not a language, assembly is a family of languages, just like LISP is also a family of languages, if all the code that you write maps directly to a instruction that a machine can do, this language can be called an assembly language, the name "assembly" comes from the fact that what the machine is executing is not exactly the instruction, the machine can execute multiple instructions at once and these instructions are assembled in to binary, the following table illustrates how that happens

So multiple assembly instructions gets assembled to one 16, 32 or 64 bit chunk of data(0000001000000100) that a computer can execute, for example the jump instruction in assembly might be represented by the last 4 bits in these chunks, the first 3 bits represents the move instructions, and so on... you got the idea, you don't need to understand any of this to write assembly, the assembler is smart enough to do it.

An assembly language is made for a specific CPU, since different CPU can do different things and execute different instructions, the difference between what a CPU can do makes the CPU more suited for different tasks, if you need to do a lot of multiplication, using a CPU that can do multiplication on hardware will be a lot better than one that can only add.

Differences Between Hardware

Depending on your task, difference instruction will be more important, and that dictates what instructions will be implemented on hardware, creating chips that can execute complex instructions is expensive but also make the execution of those instructions much faster, that's why people use GPU's for AI, mining crypto and games, GPU's have expensive instructions implemented on hardware, so things that would take a lot of instructions in a regular CPU can be done with a single GPU instruction, GPU makers release documents with these instructions so assembly creators can create a assembly language that maps to these instructions, or even higher level languages like shader languages for example, the following is a pdf document with the instructions for the AMD R700 family if you're curious https://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf

Why am I writing This?

Most of the tutorials online focus on teaching assembly bottom up, show the instruction tell what the instruction does, then show another instruction and so on, but I think this way of teaching assembly is much more complicated than it needs to be, assembly is not like other high level languages so knowing what a specific instruction do has no value for a python or Javascript developer, because that instruction doesn't map to anything that you already know, saying that MOV moves numbers to registers has no meaning to a high level language developer, the first thing that he's going to think is, what is a register and why would I move numbers to it? So instead of doing this we're going to implement a function that returns the value of a number in a fibonacci sequence, first we're going to write the code in Javascript, then we're going to translate the code to assembly and conceptually "map" each part to the high level code.

Fibonacci Code

Each number in the Fibonacci sequence is the sum of the previous two values expect for the first two values which are 0 and 1
Let's see the javaScript implementation again.

function fib(n) {
  if(n < 3) {
    return n - 1;
  }
  return fib(n - 1) + fib(n - 2);
}

fib(1); // return 0
fib(2); // returns 1
fib(3); // also returns 1 (0 + 1)
fib(4); // returns 2 (1 + 1) and so on...

Ok now let's talk about the " high level concepts" used in this implementation, the idea is the following.

name the concepts
find similar concepts in assembly
translate the concepts to assembly

Naming the concepts

The first concept is "functions", function fib().. is a
function in JavaScript, we can think about what properties does a function have, let's number the properties and think about them.

A function can be called from anywhere in the code.
You can pass data to the function.
The function have a scope.
The function can return data to the caller code execution resumes right after the function call, after the function returns, now we think about something in assembly that is conceptually similar, something that can emulate what we want to do.

Something Like A Function

There are no functions in assembly, but we have something close to it called procedures, if you think about a function in another way, a function is a jump in the program execution, in our example the first line executed was 7: fib(1) this line prompt a jump in the execution to line 1: function fib(n) { in assembly you can name a specific line and tell the program to jump to that line and keep executing the code.

global _start

_start: call Proc
        ret

Proc: push 99
      ret

Code execution starts at _start which is a procedure, the line at start and all following lines will be executed unless we have something to change the next line to be executed, in this case we call another procedure called Proc there's nothing special about the name, it's just a label that's going to be replaced by the address in memory of the next instruction to be executed.

When we run call Proc code execution will jump to line 7, ok so now we have a way of jumping between different parts of the code, the first property that we want, let's look at what is happening.

Above is the code that we just wrote being executed, the first column is the memory address of that line of code, code execution start at the first address, the address is incremented to the next address, and the code on that address is executed, except when we use call, call changes the value of the next line to be executed, so code execution jumps to the instruction push 99, the next line is ret which works like return but can not return values, after that, code execution resumes after the call instruction, ok that's one part of the puzzle now we have a way of changing the execution flow, case 1 is complete.

Passing Data To The Function

Now that we have something that we can call like a function, how do we pass data to it? most of what a higher level language do is managing memory, the biggest benefit of using lanuguages like javascript and python is that you don't need to manage memory yourself, most runtime errors are caused by bad memory allocation, either by the language, OS or the developer in most computers and OSs you have at least 3 ways of saving temporary data that is being used in a process, you have registers, the stack and the heap, registers are the easiest one to understand and use in assembly, an register is a integrated circuit that can save bits, it's hardware, and you have a limited number of those in your computers, a lot of register are used for other tasks by the CPU, so it's not safe to use them, that's why you hear people talking about general purpose registers these are the registers that're safe to be used by your running program, the following image is a list of almost all the registers in an modern computer.

Since a register is hardware, registers are global, even between different programs, if you change the value of a register in one part of your program and then call another library or procedure that reads or change this registers you gonna affect your program or the library that you're using, so it's always a good idea to keep the number of registers that you use to a minimum, just like we do with global variables, The road to programming hell is paved with global variables.

How can we save something in a register?

That's when the MOV instruction comes in, you can move a value to a register MOV RAX, 1 here we're moving the value 1 to the RAX register, and we can do that to save a value and then read it in another part of the program, which is similar to passing data to a function, we can do something like this.

global _start

_start: mov rax, 1
        call Procedure
        ret

Procedure: 
        add rax, 1 ; now the value on rax will be 2 
        ret

In the snippet above we save the value on rax, call the procedure, then read the value from that variable and do something with it, simulating passing values to a function, so 2 is also done.

Creating Scope

Another useful function property is scopes, variables created in a function only exist on that scope and if you recursively call that function passing new values, updating these values won't affect other scopes, this is a key property of functions and is the main reason why they're so useful.

function fib(x) {
  ....
  fib(2);
  ....
}

fib(1);

So can we do something like this in assembly? How can we simulate scope?

As you might have guessed we don't have the concept of a scope in modern CPUs, so we have to simulate one, if you're paying attention the first thing that you would think about would be to use the registers to save and load different scopes depending on what you're executing at that moment, the problem is that we have a very limited number of registers so if we do that we're going to run out of registers to save values pretty fast, they're global as well so abusing registers is not a good idea, OK, so the other general options to save things are the stack and the heap, let's talk about the stack.

The Stack, Another Way Of Saving Data.

Another way of saving data is the stack, the beautiful thing about the stack is it's simplicity,when you run your program the OS reserves some memory for the program to run and sets the address of the beginning of that memory to a register rsp, when you push a value to the stack, it saves this value to the current memory address saved on rsp and increases the memory address to the next one in the stack and saves it on rsp again, when you pop a value, it saves the value on whatever register you tell it to and decreases the rsp memory address number again.

We could do all of that ourselves but x86_64 comes with two instructions that already do all of that for us, push and pop, let's see that happening step by step.

This is the code that we gonna run

global _start

_start: push 99 
        push 99
        pop rax
        pop rbx
        ret

We can push a value to the stack using push, push 99 will push the value 99 to the stack, 99 is 63 in hexadecimal

And this is the stack before running anything

The first column is the memory addresses and each line is the value in memory, when we run push 99 the stack changes

Now we can see our 99 or 63 in hexadecimal there, if we run the next instruct push 99 again, 63 is added to the stack

As you can see we add to the stack in chunks of 64 bits, because my computer is 64 bits if you're using a 32 bit computer you would see a word half the size as the one we're seeing here, you can read more about what word means here here, but basically a word is the size of the data that makes sense to work on a given the computer architecture, you can work with 32 bit registers in a 64 bit computer, but that's going to make your life harder in most cases, let's keep executing our code to see what happens when we pop a value from the stack, pop rax remove the value from the stack and save that value on a register, rax in this case, that's one of the reasons why the default size of the word matters because if we tried to save a 64 bit value in a 32 bit register bad things would happen as you can imagine, rax is a 64 bit register so we can safely do that, let's see what happens to the stack after we pop that value.

Now we only have one 63 again and the value of the registers that points to the top of the stack(RSP) was decreased, if we ran the next pop we gonna remove the next value from the stack and the stack will be in the same state as it was before we started the program, RSP will point to the beginning of the stack. the OS allocates memory for the program and decides which address the stacks starts on.

Using The Stack To Save Data And Create Scope

Now that we know how the stack works, we can use the stack to save and retrieve all register values before calling another procedure, by doing that we can create scope between the different procedure calls, let's see the easiest possible example of that in practice.

_start: mov rax, 1 ; the value that we pass to Procedure
        call Procedure
        ret

Procedure: 
        push rax ; saves rax on the stack
        call ChangeParam ;call a procedure that changes rax
        pop rax ;restore rax original value from the stack 
        ret

ChangeParam:
        add rax, 1; now the value on rax is 2
        ret

Saving the values that we're working on in the stack and restoring them creates scope in the procedure, we could have done the same thing on ChangeParam that's usually the way that we do when we call external libraries, we need to restore the values before calling them, great, now we have a way of creating scope, 3 is done.

Returning Data To The Caller

Well this one is pretty easy, since everything is global in assembly, the caller just needs to read the value in the right place after the call, so if inside the procedure we save the result of the procedure in the RBX register we just need to read the value from that register after the procedure

_start: call Procedure
        mov rax, rbx ; now rbx has the result of proc
        ret

Procedure: 
        mov rbx, 99 
        ret

And with that we're done, now let's analyze the next line of javascript code and do the same thing that we done for the function.

If Statements In Assembly

if(n < 3) {

This is the next executed JavaScript, as you might imagined by now, Assembly x86 doesn't have if statements either, The main function that we want to simulate is Compare two values and based on the result of the comparison jump to different places

Conditional Logic

This one is pretty straight forward, we have CMP in assembly the way that it works is pretty simple, we compare two values, then we use a conditional jump to jump in specific cases, for example we could compare the value on RAX with 1 then do a conditional jump if they're equal.

_start:
  cmp rax,1
  je Proc

Proc:
  push 99

That's pretty much it, not much to talk about here, we have conditional jumps for each comparison greater than(JG), less than(JL), equal(JE) and for some other special cases.

Recursion

The next line is return n -1 we already implemented the return logic, we just need to save the value in the register and use ret to return from a procedure, and for n - 1 we just need to use decrement dec on the register holding the value of n, the last line of javascript is return fib(n - 1) + fib(n - 2); and we also implemented everything that that is to know about this one, the main thing about this line is the fact that is recursive, and we already implemented the idea of a function with scope, so we only need to call the procedure save the data on the stack and the rest that we did to simulate a function call, Now we can put everything together and implement the solution.

Putting Everything Together

Let's go line by line

Function definition

function fib(n) {

Procedure definition

Fib:

If statement

if(n < 3) {

Compare and jump
rax holds the parameter passed to the function

cmp rax,3 
jl L3 ; jump to L3 if rax is less than 3

Inside the if block

return n - 1;

We jump to another part of the code, decrement the register and save the value to another register before returning, since this is the last case we don't need to push rax to the stack again, jumps works in a different way than procedure calls. when you call a procedure ret gonna return you in the line that you called that procedure, when you jump it can look like a procedure but it isn't, the code belongs to the procedure where you used the jump so ret is going to return from that procedure, it works just like a if statement in this case.

L3:
     dec rax
     mov rbx, rax
     ret

return fib(n - 1) + fib(n - 2);

This one is complicated

push rax ; saves the value of rax, rax is n in this context
dec rax  ; decrements rax 
call Fib ; call procedure now with the modified rax value
pop rax  ; pops the original value of rax from the stack
push rbx ; push the result from the previous Fib call 
dec rax  ; decrements rax two times ` n - 2`
dec rax
call Fib ; calls procedure now with the modified rax value
pop rcx  ; pop the result from the *first* Fib call
add rbx, rcx ; add the result from the first Fib call with the second one
ret

As you can see simulating scope is the trickiest part but it's not that hard if you take some time to think about it, we're just using the stack to save stuff that we want to use later and not change the things that we're working right now, the stack is used a lot in this manner, "you don't want to use right now but needs this value later? put it on the stack"

Final Code

global _start
_start:
    mov rax,5
    call Fib 

    ; Some boilerplate code to return control to the OS
    mov eax,1       ; Code for Exit Syscall
    mov ebx,0       ; Return a code of zero 
    int 80H             ; Make kernel call
    ret

Fib:
    cmp rax,3
    jl L3 
    push rax
    dec rax
    call Fib
    pop rax
    push rbx
    dec rax  
    dec rax
    call Fib
    pop rcx
    add rbx, rcx
    ret

L3:  
    dec rax
    mov rbx, rax
    ret

in the case above 5 is the parameter that the fib function receives so the returned value should be 3, 3 is the fifth element in the fibonacci sequence.
if we ran our code and inspects the value of rbx at the end of the execution we get 3

IO In Assembly

One thing that is clearly missing is the ability to pass arguments to our program and to print out the results, I didn't touch on this before because is not really assembly, but we can do that to make our program more useful, the way that something like print("hello world") works is by calling a function of the Kernel that knows how to print things on the screen, we also need to read the arguments passed to the function, transform the numbers in to characters and the characters in to numbers by offsetting the values in the ascii table , I'm not gonna explain this here because the post is already pretty big, the following is the code with IO, you can pass the index in the fibonacci that you want, and the code returns the value in the sequence.

Final Code With IO

%macro  ExitProg 0
    mov eax,1       ; Code for Exit Syscall
    mov ebx,0       ; Return a code of zero 
    int 80H         ; Make kernel call
%endmacro

section .data
    newline db 10,0

section .bss
    printSpace resb 8
    digitSpace resb 100
    digitSpacePos resb 8

section .text

global _start
_start:
    mov rdi, qword [rsp + 16] ; Get the third item in the stack, which is the first argument passed to the program
    movzx rax, byte [rdi]   ; Get the first character 
    sub rax, 48             ; Convert from ASCII to decimal
    call Fib                ; after call Fib `rbx` has the result
    mov rax, rbx            ; moves the result to rax
    call printRAX           ; call procedure to print the result
    mov rax, newline        ; move new line to rax
    call print              ; prints the new line
    ExitProg

Fib:
    cmp rax,3
    jl L3 
    push rax
    dec rax
    call Fib
    pop rax
    push rbx
    dec rax  
    dec rax
    call Fib
    pop rcx
    add rbx, rcx
    ret

L3:  
    dec rax
    mov rbx, rax
    ret

print:
    mov [printSpace], rax
    call printLoop
    ret

printLoop:
    mov cl, [rax]
    cmp cl, 0
    je endPrintLoop
    inc rbx
    inc rax
    jmp printLoop

endPrintLoop:
    mov rax, 1 
    mov rdi, 0 
    mov rsi, [printSpace]
    mov rdx, rbx
    syscall
    ret

printRAX:
    mov rcx, digitSpace
    mov [digitSpacePos], rcx

printRAXLoop:
    mov rdx, 0
    mov rbx, 10
    div rbx
    push rax
    add rdx, 48

    mov rcx, [digitSpacePos]
    mov [rcx], dl
    inc rcx
    mov [digitSpacePos], rcx

    pop rax
    cmp rax, 0
    jne printRAXLoop

printRAXLoop2:
    mov rcx, [digitSpacePos]

    mov rax, 1
    mov rdi, 1
    mov rsi, rcx
    mov rdx, 1
    syscall

    mov rcx, [digitSpacePos]
    dec rcx
    mov [digitSpacePos], rcx

    cmp rcx, digitSpace
    jge printRAXLoop2
    ret

How To Debug Assembly

You can use gdb to debug Assembly but you need to configure it properly, this is the configuration that I use, besides that , to start debugging run

gdb fibonacci

if you needs to pass parameters to the program

gdb --args fibonacci 5

after that you can break on the _start procedure

b _start

run the program

run

Then you can use nexti to go to the next instruction and stepi to step inside a procedure.

i r <register_name> shows the value of a register e.g i r rax

We can print address values of registers and symbols with x/x $rsp to print in hexadecimal and x/d to print as a decimal value.

Code

All code is available here

Multiprocessing communication with FIFO queues

Marcell Cruz — Wed, 17 Aug 2022 16:06:02 +0000

This is part of a series of posts talking about FIFO and techniques for IPC(inter process comunication) today I'm gonna explore IPC between parent and child processes for asynchronous programming, one problem that arises when you fork a process is communication between the parent process and the child process or between child processes, usually this is solved in the standard library of the language or a third party library, node's exec and spawn for example handles communication between the process for you using something very similar with what we're gonna do, But how does that work and what can we do if we want to have more control over it, Let's explore some options.

Single Queue Multiple Processes Without Error Handling
Single Queue Multiple Processes With Error Handling

Simple Queue Spawning Multiple Processes Without Error handling

The code

I always like to start with the code, because reading the code will give you a general idea of what's going on, the following is very simple ruby code, it's written in Ruby but the idea can be used in any language, and the point is the idea not the implementation.

read_from_fifo reads from the fifo file line by line, push_to_fifo writes a line to the fifo file and process_queue reads from the input file forks the process passes the url that was read from the fifo file to the child process, the child process makes a http request and writes the result to an output fifo file, since process_queue is a recursive function that runs forever, any new line added to the fifo file will be processed.



require 'uri'
require 'net/http'
require 'json'

OUTPUT_FIFO = './output.fifo'
INPUT_FIFO = './input.fifo' 
unless File.exist?(OUTPUT_FIFO)
  File.mkfifo(OUTPUT_FIFO)
  File.chmod(0666, OUTPUT_FIFO)
end

unless File.exist?(INPUT_FIFO)
  File.mkfifo(INPUT_FIFO)
  File.chmod(0666, INPUT_FIFO)
end

def read_from_fifo(fifo) 
  rpipe = File.open(fifo, 'r+')
  res = ""
  while (r = rpipe.read(1)) != "\n"
    res << r
  end
  res
end

def push_to_fifo(endpoint, fifo)
  wpipe = File.open(fifo, 'w+')
  wpipe.write "#{endpoint}\n"
  wpipe.flush
end

def process_queue()
  rpipe = File.open(INPUT_FIFO, 'r+')
  url = ""
  while (r = rpipe.read(1)) != "\n"
    url << r
  end
  fork do
    puts "Processing: #{url} PID: #{Process.pid}"
    uri = URI(url)
    res = Net::HTTP.get_response(uri)
    puts JSON.parse(res.body)['quote']
    push_to_fifo(JSON.parse(res.body)['quote'], OUTPUT_FIFO)
  end
  process_queue
end

push_to_fifo('https://api.kanye.rest/', INPUT_FIFO)
push_to_fifo('https://api.kanye.rest/', INPUT_FIFO)
push_to_fifo('https://api.kanye.rest/', INPUT_FIFO)

process_queue()

Above is a terminal running the program, and below is another terminal where I pushed the url to the fifo file, the following is my attempt at drawing what is happening in more detail

Simple Queue Spawning Multiple Processes With Error Handling

One thing that was missing from our last implementation was error handling, if something goes wrong in the process that tries to make the GET request, the message is lost, the process is going to crash or hang forever we're not gonna try again or get any response.

We want to improve the resilience or our application and make it more fault tolerant.

What we can do in this situation is create another queue to process errors, also called a dead letter queue, any time we have a error processing a message we push the message to the dead letter queue to be processed later.

The Code

To do that we just need to push the error to the dead letter queue and to also process the dead letter, to do add we add another process just to process the dead letter queue, both processes, the main that processes the input queue and the process responsible for the dead letter queue run at the same time and wait for messages to get to the queue.



require 'uri'
require 'net/http'
require 'json'

OUTPUT_FIFO = './output.fifo'
INPUT_FIFO = './input.fifo' 
DEADLETTER_FIFO = './deadletter.fifo' 
unless File.exist?(OUTPUT_FIFO)
  File.mkfifo(OUTPUT_FIFO)
  File.chmod(0666, OUTPUT_FIFO)
end

unless File.exist?(INPUT_FIFO)
  File.mkfifo(INPUT_FIFO)
  File.chmod(0666, INPUT_FIFO)
end

unless File.exist?(DEADLETTER_FIFO)
  File.mkfifo(DEADLETTER_FIFO)
  File.chmod(0666, DEADLETTER_FIFO)
end

def read_from_fifo(fifo) 
  rpipe = File.open(fifo, 'r+')
  res = ""
  while (r = rpipe.read(1)) != "\n"
    res << r
  end
  res
end

def push_to_fifo(endpoint, fifo)
  wpipe = File.open(fifo, 'w+')
  wpipe.write "#{endpoint}\n"
  wpipe.flush
end

def process_deadletter() 
  puts "Waiting for messages in the dead letter queue"
  rpipe = File.open(DEADLETTER_FIFO, 'r+')
  url = ""
  while (r = rpipe.read(1)) != "\n"
    url << r
  end
  fork do
    begin
      puts "Dead Letter Processing: #{url} PID: #{Process.pid}"
      uri = URI(url)
      res = Net::HTTP.get_response(uri)
      puts JSON.parse(res.body)['quote']
      push_to_fifo(JSON.parse(res.body)['quote'], OUTPUT_FIFO)
    rescue => e
      puts "error: #{e}"
    end
  end

  process_deadletter
end

def process_queue()
  puts "Waiting for messages in the process queue"
  rpipe = File.open(INPUT_FIFO, 'r+')
  url = ""
  while (r = rpipe.read(1)) != "\n"
    url << r
  end
  fork do
    puts "Processing: #{url} PID: #{Process.pid}"
    begin
      uri = URI(url)
      res = Net::HTTP.get_response(uri)
      puts JSON.parse(res.body)['quote']
      push_to_fifo(JSON.parse(res.body)['quote'], OUTPUT_FIFO)
    rescue => e
      puts "Error trying to process #{url} sending message to deadletter queue"
      push_to_fifo(url, DEADLETTER_FIFO)
    end
  end
  process_queue
end

fork do
  process_deadletter
end

process_queue

If we run the previous code we get the following response

Now we have two processes waiting for messages in the queue,
if we append a message to the input.fifo file with



echo 'https://api.kanye.rest/' >> input.fifo

We get the response

Nothing new here, we pass a valid url and everything worked fine, but if we pass a invalid URL we can see the dead letter queue working



echo 'invalid url' >> input.fifo

The first process fails and send the message to the dead letter queue, the dead letter queue process starts processing the new message in the queue, it also fails since in this case it's a invalid URL

Adding A Delay To The Second Call

In a real situation we would want to add a delay to the second call, if the service that we're hitting is not working now, it's unlikely that it's going to be working some milliseconds later, what we want to do in this case is to add a delay to the process that handles the dead letter queue, we have a lot of different approaches to do that, we can increase the time we do the retry for every retry, we can filter out some errors like the invalid URL case, since it's never going to work in this case.

In the next post I'm gonna experiment with other ways of setting up the queues and processing the messages.

Unix Domain Socket(IPC), how to make two programs communicate with each other

Marcell Cruz — Tue, 14 Jun 2022 16:48:45 +0000

TLDR;

If you're already familiar with unix domain sockets and just want a working example take a look here

This is part 2 of a series of posts exploring techniques on how to make two different programs written in different languages send messages to each other, also known as IPC(inter-process communication).

In the previous post that you can find here I talked about using Unix piping to manage the communication, now we're going to use the networking capabilities of the system to do it.

Most people know how a TCP connection works, we're not going to use TCP here but it follows the same pattern, which basically goes as follows, Computer 1(The Server) creates a sockets and binds this socket to a specific port for new connections, Computer 2(The Client) also creates a socket and connects to this port, a connection is established and they can send and receive messages to each other, we can make a analogy with regular physical sockets here, the established connection is the cable, the sockets are sockets, and the port is the location of the socket.
All this is well and good, but it's also a little bit too much when both programs are running in the same computer, When the programs are running in different machines you need a lot of different protocols layers to establish this connection, you need routers, the Ethernet protocol, internet protocol etc..., but in our case we just want to establish communication between two programs running in the same machine, that's a perfect use case for Unix domain sockets, We can use the same process and methods that we use to connect two different computers to connect two different programs.

The Client

In this case the client is in Ruby and the server in C, the following is the code for client



require "socket"

close = false 
socket = UNIXSocket.new('/tmp/mysocket')
while close == false do
  puts ">"
  message = gets
  begin
    socket.write(message)
    response = socket.readline
    puts response
  rescue Errno::EPIPE
    puts "lost server connection!"
    close = true
    socket.close
  end
end

The main difference here is that instead of binding the socket to a port we bind the socket to a file, the rest of the process is pretty much the same, create the socket send a message to the counter part, wait for a answer, close the connection or not depending of what you want to do, in the code above we only close the connection if the lost connection to the server otherwise we keep sending messages to the servet.

The Server

The server code is pretty much the same thing, is much bigger since it's written in C, and we're managing any possible errors.



#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/socket.h>
#include <sys/un.h>
#include <sys/types.h>

static const char* socket_path = "/tmp/mysocket";
static const unsigned int nIncomingConnections = 5;

int main() {
  //create server side
  int s = 0;
  int s2 = 0;
  struct sockaddr_un local, remote;
  int len = 0;

  s = socket(AF_UNIX, SOCK_STREAM, 0);
  if( -1 == s ) {
    printf("Error on socket() call \n");
    return 1;
  }

  local.sun_family = AF_UNIX;
  strcpy(local.sun_path, socket_path);
  unlink(local.sun_path);
  len = strlen(local.sun_path) + sizeof(local.sun_family);
  if(bind(s, (struct sockaddr*)&local, len) != 0) {
    printf("Error on binding socket \n");
    return 1;
  }

  if( listen(s, nIncomingConnections) != 0 ) {
    printf("Error on listen call \n");
  }

  int bWaiting = 1;
  while (bWaiting) {
    unsigned int sock_len = 0;
    printf("Waiting for connection.... \n");
    if( (s2 = accept(s, (struct sockaddr*)&remote, &sock_len)) == -1 ) {
      printf("Error on accept() call \n");
      return 1;
    }

    printf("Server connected \n");

    int data_recv = 0;
    char recv_buf[100];
    char send_buf[200];
    do{
      memset(recv_buf, 0, 100*sizeof(char));
      memset(send_buf, 0, 200*sizeof(char));
      data_recv = recv(s2, recv_buf, 100, 0);
      if(data_recv > 0) {
        printf("Data received: %d : %s \n", data_recv, recv_buf);
        strcpy(send_buf, "Got message: ");
        strcat(send_buf, recv_buf);

        if(strstr(recv_buf, "quit")!=0) {
          printf("Exit command received -> quitting \n");
          bWaiting = 0;
          break;
        }

        if( send(s2, send_buf, strlen(send_buf)*sizeof(char), 0) == -1 ) {
          printf("Error on send() call \n");
        }
      } else {
        printf("Error on recv() call \n");
      }
    } while(data_recv > 0);

    close(s2);
  }

  return 0;
}

Don't be intimidated by the code, the code is as simple as the Ruby one, the main different is that we first create a socket then we bind the socket to an address just like we do for the internet socket, but in this case the address is the file location and not a port, the main reason why the C code is that much bigger is because of all the string manipulation and memory management that we need to do which we get for free in Ruby, under the hood Ruby is using the same libraries to create the socket written in C, if we run the client and the server we can send messages from the client to the server and then get the server response which in this case is the same message.

The top terminal is running the server and the bottom one the client, we type hello! in the client and hit enter to send the message to the server, we could close the connection after sending the message but we kept alive that's why we have a while loop running.

FIFO and Bidirectional piping vs Unix Domain Socket

I've already talked about FIFO and bidirectional piping here , Both of the solutions are used for IPC(inter-process communication), they both use FIFO but through different mechanisms.
The main different from an user perspective is reliability, while creating FIFO and piping data is very raw, hacky and easy to understand, it's not very controlled and reliable.

Using Unix domain sockets we can leverage all OS level functions live recv, bind, send and connect, all these functions throw errors so we have a much more controlled way of writing code that works, but if you just want to script some solution for your own personal use I don't see a problem using FIFO directly like in the bidirectional piping example, you just have to keep in mind that reliability and portability might suffer or even don't work at all if you try to run it in a different system.

Bidirectional piping, how to make two programs communicate with each other

Marcell Cruz — Wed, 23 Feb 2022 19:41:41 +0000

First of all a small refresher on what piping is and how it works
when you do something like



echo 2 | cat

stdout of echo is being piped to cat's stdin, cat writes again the result that was read from stdin to stdout and you see the result in your terminal, this is a example of unidirectional piping, bidirectional piping would be if cat could write back to echo and echo could read what was piped, basically you would have two program that can communicate with each other.

On Linux we do this creating a named pipe, which is just a special type of file using the FIFO(first in first out) method so one program can write to that file and the other can read and vice versa, the two easiest ways of creating a FIFO is using the program mkfifo, or using bash built-in function coproc, both do basically the same thing the only different is that when using coproc you end up with more elegant code depending on who you ask, let's create a simple program and implement bidirectional piping using mkfifo, which is more explicit, I'm gonna write one program in bash and the other one in javascript(node) but any language that can write to stdin, stderr and read from stdout would work(so basically all of them)
let's create three programs, prog1, prog2 and program to create the FIFO and call the other programs

prog1.sh



echo "bash: hello!"
read line
echo $line 1>&2
echo "bash: how are you doing?"
read line
echo $line 1>&2

prog2.js



const readline = require('readline');

const rl = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
  terminal: false
});

rl.on('line', function (cmd) {
  if(cmd == 'bash: hello!') {
    console.error(cmd);
    console.log("javascript: hello!");
  }
  if(cmd == 'bash: how are you doing?') {
    console.error(cmd);
    console.log("javascript: I'm fine, thanks!");
  }
});

and the program to create the FIFO and call the two other programs

start.sh



mkfifo p1 p2 
bash ./prog1.sh > p1 < p2 &
node ./prog2.js < p1 > p2 

rm -rf p1 p2

We delete the FIFOs in the last line to avoid a warning the next time we run bash start.sh, the warning would just tell us that the FIFO already exist.
now we can run everything with



bash ./start.sh

It works!!! one program can send messages to the other program, when both programs end, start also ends, also we can only read or write to the FIFO if two program open the FIFO file, if you remove the last line of start.sh you can see the FIFO files that were created in the same directory.

If you try to open the FIFO file in a text editor, your text editor will probably crash on hang indefinitely. you might think that the same thing is possible just creating regular files, but the code would be a lot more cumbersome, FIFO allows us to write pretty elegant code connecting two different programs, or two different instances of the same program, let's talk a little bit about the actual code



mkfifo p1 p2

This command creates two FIFO files called p1 and p2



bash ./prog1.sh > p1 < p2 &

here we run the first bash program and tell it to point it's stdout to p1 and stdin to p2



node ./prog2.js < p1 > p2

here we run the second program and do the opposite, we tell him to read from p1 and write to p2



rm -rf p1 p2

This line is not required it just avoids the warning when we run mkfifo again the next time

Now let's have a look at the programs themselves, first the bash program



echo "bash: hello!"
read line
echo $line 1>&2
echo "bash: how are you doing?"
read line
echo $line 1>&2

The only think that might be confusing here if you don't know bash very well is the line



echo $line 1>&2

here we're redirecting from stdout to stderr if we don't do this we can't see the messages in our terminal, remember that the stdout is going to the other program and not to the terminal, if we just echoed without the redirecting the string will go to the FIFO and be read by prog2, same thing for the javascript code



console.error(cmd);

we do the above so we can print to stderr, the javascript program prints the bash messages and the bash program prints the javascript messages, another difference is that the javascript program calls the same callback every time it reads something from the fifo that's why we have the if statements to identify which message was receive and reply accordingly.

Supply, Demand, Price and Game Theory

Marcell Cruz — Mon, 10 Jan 2022 15:34:16 +0000

How price can be determined with game theory.

How game theory and simple math can be used to calculate the price of something and how this model is super powerful to explain a lot of others interactions.

Before using game theory to explain the price, I'm gonna do something that I like to do when I'm trying to understand something, and that is trying to explain it by myself without reading anything else, so let's try to explain how the price of something can be decided first only using our intuition so we can start forming a picture of what we are talking about, then it will be easier to understand the math that tries to model the behavior.

You want to charge for a service or product, to simplify, let's say it's a service that virtually only takes time,
to charge for it you need a value that will cover your time spent doing it at least, so let's say your cost of living is 1000 USD/month, if you can perform this service 100 times per month then you can charge no less than 10 USD otherwise you can't pay for your cost of living, besides that you probably
want to have a positive balance, meaning that you earn more than you spend so you want to charge more than the price that would be enough for you to survive.

So one thing that we can say about charging for something is that the cost of doing it(time and resources) is important.
because it dictates the minimum price possible, time is important here, because eventually bills will have to be paid, and these bills must cover the time frame that you took to do the work otherwise your would spend more than you earn which is also viable if you have debt, hoping that in the future it can be paid, but let's simplify things and assume you can't spend more than you earn.

With just these observations we can say that the amount of times that you do the same service or sell the same product is also important, and these two things are related, the longer it takes to do something the more you have to charge for it and you're going to be doing that less over a fix amount of time.

On the contrary the cheaper something is, the more times you have to do it, those two things balance out, here we're talking about a service but it could also be a product, so the next question is how can we describe this relation?

In our case the price is 10, because we think we can do it 100 times over a month and our cost of living is 1000, if we can increase the times that we do it we can reduce the price,
so let's make a table and think about this relation.

price   quantity
10      100
20      50
40      25

The product is the same for all these rows, p x q = 1000
because 1000 is our minimum. We can also think about the advantages and disadvantages of each relation, of course this is a much more complicated subject to think about, but we can at least intuit that in general something cheaper will be easier to sell that something more expensive(demand curve, also called the law of demand) so increasing the quantity and lowering the price is generally speaking a good idea, this also means increasing efficiency, that's specially true in a very competitive market.

So now that we have a pretty good idea of the pieces that
make the price of something(quantity, time, cost) we can expand our view and start to think about other things that influence the price.

Until now we only took the point of view of the seller, but to sell something you also need a buyer, someone willing to pay the price that you're asking, assuming this buyer is a rational actor, he wants to pay the least amount possible for the product or service, everything else being equal.

We already know how to calculate the minimum price, but just selling something for the minimum possible price is not a good idea since you could earn a lot more if your buyer is willing to pay more, so now let's try to intuit what influences the upper limit, the maximum price that you can sell something, assuming two companies c1 and c2 both located at the same city and selling the same product, what will determined the price of the product in this situation?

We can say that our buyer being a rational actor and knowing that both product are mostly the same will choose the lowest price, but neither company wants to sell for the lowest price possible, not only that but since we certainly have multiple potential buyers in the same city, the buyers also will compete between themselves to buy the product or service it's a thug of war.

Now that we have a good intuition about what's happening and the pieces that matter, let's define a demand function and think about the interaction between the parties using some game theory.

Iterated Elimination Of Strictly Dominated Strategies Convergence

This long and complicated name is actually not that hard to understand, let's break each part and think about it.

Iterated Elimination

We called iterated elimination because we gonna remove the strategies that the companies will not play, since we assume they are rational actors and those strategies will be sub-optimal, it's iterated because we do it in rounds and each round influences the next.

Strictly Dominated Strategy

An strictly dominated strategy is an strategy that isn't the best strategy in any situation, it can be better than other strategies on a particular situation if you have more than 2 choices but it's never "the best strategy", We gonna eliminate these strategies in each iteration since there's a better strategy to be played.

Convergence

Each strategy elimination, which in this case means the quantity that the company should produce will lead to a smaller price window, this price window will get smaller as we eliminate more and more strategies until we end up with just one optimal quantity to be produced by the companies.

Applying the convergence

Now That we have a idea of what we're going to do, let's do it.

Given a demand function

d(q) = 100 - (q1 + q2)

Where 100 is all meaningful data that influences this function besides the quantity and the price, q1 is the quantity that company 1(c1) will produce and q2 is the quantity that company 2(c2) will produce, the objective of each company is to have the highest profit possible, We can calculate the profit based on the demand function, the quantity being produced and the cost of producing the product.

The profit payoff function of firm 1

              [demand function]   [production cost] 
v1(q1, q2) = (100 -  q1 - q2)  q1 -     10q1

In a very competitive market two companies being rational actors can only influence the quantity that they produce, the demand multiplied by the quantity produced by company 1(c1) will give us the maximum possible revenue of c1, removing the cost of production will give us the profit of c1
we can rearrange the equation.

90q1 - q1**2 - q1q2

Now we can use the derivative at 0 to calculate the max profit possible.

The derivative being

90 - 2q1 - q2 = 0
//after rearranging
q1 = (90 - q2)/2

We end up with a function that will tell us the quantity that we need to produce based on the quantity that our competitor produced, if our competitor produce 0 units we have.

q1 = (90 - 0)/2
q1 = 45

So 45 is the ideal number of units to maximize our profits if our competitor don't produce anything.
Now something interesting happens, if our competitor knows that we know that 45 is the ideal number if he doesn't produce anything, he knows that we'll never produce more than 45, since producing more than 45 will be a dominated strategy, S1 = (0, 46) is worst than S2 = (0, 45),

Knowing that, he can calculate the minimum amount ideal for him, he just needs to plug 45 in his equation.

q2 = (90 - 45)/2
q2 = 22.5

What that means is that the quantity floor is 22.5, in other words a strategy that produces less than 22.5 is also a dominated strategy, because S1 = (22.5, 45) is better than
S2 = (22, 45) and we know that the upper quantity for company 1 is 45 so that company will never produce more than 45.
So now we also have a lower floor, we have a quantity window between 22.5 and 45 anything outside this window is a bad idea.
But it's not over yet we can now apply the same idea
for the floor 22.5, now company 1 knows that company 2 will never produce less than 22.5 we can plug this value in our function and figure out another quantity ceiling.

q1 = (90 -22.5)/2
q1 = 33.75

since we know that company 2 will never produce less than 22.5 we should never produce more than 33.75.

Do you see where this is going? now we can "iterate" again, with a new upper ceiling of 33.75

q2 = (90 - 33.75)/2
q1 = 28.125

and again...

q2 = (90 - 28.125)/2
q1 = 30.93

If we keep doing this we can see that the value goes to 30, which is the best possible strategy for both companies, so the quantity that they should produce to maximize profit assuming both companies are rational actors is exactly 30, let's plug this value in to the profit function and see what happens.

v1(30, 30) = (100 -  30 - 30)30 - 10 * 30 
v1(30,30) = 2100

So both companies profit should be 2100 and the price of each unit 80, if another company would join the market the profit and price would drop for all of them, if we kept
adding more competitors eventually the price would drop to the cost of production and nobody would make a profit, considering that the demand didn't change

Game Theory is a interesting exercise to think about things that happens in the real world and also serves as a guide line on how things "should look" but the reality is that the real world is far more complicated, for starters there're no "rational actors" in the real world, people and companies act irrationally all the time, we usually don't have all the data and when we do it might not be correct so that magic "100" that we considered in the demand functional for "all the things that influence demand" can be wrong when
you apply to a real situation, nevertheless none of this should discourage you to try to understand things, model them and apply what you know to real life situations, you might not have the perfect answer but you can at least understand how things should look in a ideal situation which is usually not very far from the truth.

How Unix commands works

Marcell Cruz — Thu, 29 Jul 2021 16:39:09 +0000

If after using these commands for years you ever wondered how they work, this is a high level exploration of someone who just had the some thoughts some days ago, this is the result of my exploration, I hope you learn something from it and get more interested in OS development in general, first of all, a little bit of history, all theses programs are originally part of the gnu os, the first atempt to create a free os, they were written in the mid eighties and are older than Linux itself, they were developed for unix than ported to Linux and are posix compliant, ls in particular is part of coreutils, a separate module of the gnu os, you can see coreutils source code here, Most of gnu os was developed by Richard Stallman, the original neckbeard

GNU/Coreutils is used by most Linux distribution, and you can even use it on Mac OS and Windows, in Mac Os you can even replace the propietary coreutils. As you might know, Linux is not a OS, Linux is just a kernel, you need higher level programs to use your system, that's why people sometimes refer to Linux as GNU/Linux, because it's the kernel with the other utilities necessary to realistically use the system in your day to day activities, otherwise you would have to implement everything yourself. I'm gonna focus on how ls is implemented on top of the kernel, if you wanna know what happens right after you type ls in your terminal you should read this amazing explanation.

How ls works

You can see ls source code here,
this blog posts focuses on explaining ls, but all other core utilities work in a similar fashion, let's get to it.

the core functionality of ls are actived using two lower level functions opendir and readdir, opendir opens a directory, which is really just a file in Linux like everything else and readdir reads this file line by line, each line as you might have guessed is the directory contents, plus . and .. which is used by the system to reference this directory and where this directory is, if you search for these functions in src/ls.c you would find this

...
DIR *dirp;
struct dirent *next;
dirp = opendir (name); // line 2918
...

for opendir

...
while (1)
    {

      next = readdir (dirp); // line 2988
      if (next)
...

for readdir

As you might have guessed this while loop is ls's core functionality, it's the part that reads the file entries in the dir and prints them out, or do whatever it needs to do depending on the options that you pass to ls, all other stuff in the source code is error handling, parsing and applying options, ls has a ton a options that accumulated over the years, if we remove all these options we could implement ls with just a few lines of code, so let's do that

#include <stdio.h>
#include <dirent.h>

int main()
{
    DIR *folder;
    struct dirent *entry;
    folder = opendir(".");
    if(folder == NULL)
    {
      return(1);
    }
    while( (entry=readdir(folder)) )
    {
        printf("%s ",
                entry->d_name
              );
    }
    printf("\n");
    closedir(folder);
    return(0);
}

if we compile and run our ls version, we get this

as you can see the output it's pretty similar to gnu ls already, the only difference in this case is that ls is color coding different file types in my system and ls by default don't display . and ...

Ok, now we have a pretty good understanding of how ls is implemented and the main libraries that it uses("stdio.h" and "dirent.h"), but where does these libraries come from? we need to dig deeper.

glibc and system calls

stdio.h and dirent.h and pretty much anyother c library installed in your system lives in /usr/include, stdio and dirent in particular are installed when you install glibc, yet another gnu project, printf comes from stdio, opendir and readdir comes from dirent, just like ls these functions are abstraction to make it easier to do the things that you want to do in your system, actually everything is a abstraction all the way down

ls is a application that uses glibc, glibc itself make some system calls

as you can see in the graph we don't necessarily need to use glibc to implement ls we could make the system calls ourselves, let's do that!

#include <linux/fs.h>

...
 u8 Buff [50];
 int fd;
 memset (Buff, 0x00, sizeof (Buff));
 mm_segment_t old_fs = get_fs ();
 set_fs (KERNEL_DS);
 fd = sys_open ("/etc/Info", O_RDONLY, 0);
 if (fd> = 0)
 {
  sys_read (fd, Buff, 50);
  printk ("string:% s/n", Buff);
  sys_close (fd);
 }
 set_fs (old_fs);

Kernel level and User level

The ls clone that we wrote is running on user level, all the functions that we used, opendir, readdir and printf are user level functions, a lot of things are restricted at user level, functions at user level call other functions at kernel level, this is what we call a "system call" so the function opendir for example eventually will call sys_open, which is a system call that it self calls do_sys_open which is a kernel level function, the kernel is actually what comunicates with devices and the cpu

The kernel is a program that is always in memory and facilitates the comunication between the hardware and applications, one of these utilities is the functions that we just used, we could go deeper and explore how do_sys_open is implemented at kernel level, but that's propably another post

As you can see it's a lot harder to understand the sys_call version and you do very little with a lot of code, as you get closer to the kernel you have to deal with internal calls of the system, that's the reason why the kernel exists, it's a abstraction on top of the machine so you don't have to deal with all this low level stuff, you can see in the code that we have to make a lot of different system calls just to read a directory, if you wanna go deeper and learn even more about how linux works I recommend this book this book I'm reading it and it's pretty good

DEV Community: Marcell Cruz

Web Essentials PART 2

Space Repetition With Anki Cards

Web Essentials

Why does Memorization matter?

The List

High Level Programming Language(Ruby, Javascript, Python, Elixir)

SQL

HTML Forms

HTML/CSS

Frontend and backend frameworks

Linux Basics

Docker

nginx

[Boost]

"ngrok" from scratch

Sibelius Seraphini for Woovi ・ Jan 28

Basic methods for comprehension

Atomistic Approach

Holistic Approach

Combination of Both Approaches is Usually the Best

Atomistic

Holistic

Combination of Both Approaches is Usually the Best

How To Create An UI Menu In Neovim

How To Write A Programming Language

The Grammar

Composing Grammar Rules

The $$ Operator And Empty Grammar Rules

Rule Types

Parsing The Language

AST

Executing The AST

Assembly For JavaScript Developers

Why Should You Care?

Lines Of Code vs Instructions

Assembly Is Not A Language?

Differences Between Hardware

Why am I writing This?

Fibonacci Code

Naming the concepts

Something Like A Function

Passing Data To The Function

Creating Scope

The Stack, Another Way Of Saving Data.

Using The Stack To Save Data And Create Scope

Returning Data To The Caller

If Statements In Assembly

Conditional Logic

Recursion

Putting Everything Together

Final Code

IO In Assembly

Final Code With IO

How To Debug Assembly

Code

Multiprocessing communication with FIFO queues

Simple Queue Spawning Multiple Processes Without Error handling

The code

Simple Queue Spawning Multiple Processes With Error Handling

The Code

Adding A Delay To The Second Call

Next post

Unix Domain Socket(IPC), how to make two programs communicate with each other

TLDR;

The Client

The Server

FIFO and Bidirectional piping vs Unix Domain Socket

Bidirectional piping, how to make two programs communicate with each other

Supply, Demand, Price and Game Theory

Iterated Elimination Of Strictly Dominated Strategies Convergence

Iterated Elimination

Strictly Dominated Strategy

Convergence

Applying the convergence

How Unix commands works

How ls works

glibc and system calls

Kernel level and User level