DEV Community: Paul DiGian

Linux Processes

Paul DiGian — Tue, 11 Aug 2020 15:17:43 +0000

Understanding the concept of processes is fundamental when working with computers.

Most articles tackle the issue from a quite shallow perspective or the wrong point of view. We will try to study them from what I believe is the most useful point of view for a senior computer engineer.

What processes are

Processes are a computational unit.

This definition is neither precise nor completely correct, but it helps in understanding.

Whenever you run a program a new process is created. Each software that you are running now in your machine is a process.

Your browser? It is (at least one) process. Your mail client? Another one. Your bash terminal? Yet another process.

Isolation

The most important concept to understand about a process is "isolation". Processes are isolated one from another. One process cannot read nor write memory that belongs to another process. Actually, it is even more strict. One process can write and read, only the memory that belongs to itself.

The difference is that a process cannot access memory that belongs to no-one. It can access only its own memory.

If the isolation is not respected and a process tries to read or write (access from now on) memory that does not belong to itself, the process is forcefully terminated by the operative system with a segmentation fault.

All the time a segmentation fault is thrown, it means that the process did not behave correctly and it was trying to access memory that didn’t belong to it.

Isolation is the greatest advantage of processes. It assures that the content of the memory never changes under your feet and that you are the only one to have access to the memory.

While this advantage doesn’t seem particularly interesting, and maybe also bland, modern (complex) software takes full advantage of it.

For instance, Redis (a NoSQL database) use a single execution unit to avoid coordination keeping the software extremely fast. The alternative would have been to use thread, hence more execution units, and to introduce locks and a great deal of complexity in managing multiple execution units changing a shared memory.

Modern browsers, have one process for each tab. This avoids malicious code to gain access to data in other tabs. The use of multiple processes, one for each tab, makes it impossible that javascript code running in a tab from a pishing website, can access any data from a tab from your bank.

Isolation is a wonderful capability, but it comes at a cost. Software, to be useful, need to read input and produce output. So a process needs some way to get input from the outside world and to push its output.

Introducing IPC or Inter Process Communication

Inter Process Communication

We discussed how processes are isolated, we agree that isolation is a great capability for processes, but that it comes at a cost. It makes difficult input/output. We are now introducing how processes can communicate with each other and with the outside world.

We will have a different section for each of the IPC methods we are describing, so the discussion on this page will be rather short.

For maintaining isolation is important that it is always the process to drive the I/O. It can not happen that some data change without the process having asked it explicitly. This is in stark contrast to what happens with threads.

Using threads it is possible that some value changes without one thread explicit permission. So a thread could read a variable, do some other work without ever writing anything to that variable, and then find the initial variable changed. Another thread, in the same process, changed that variable. This is a source of many, very complex to reproduce and to address bugs. And this does not happen between processes. Of course, there is an exception to this general rule.

We will now explore the most common IPC methods, starting from the most useful.

Sockets

Sockets are extremely important IPC methods. They allow communication between processes that reside on different machines or on the same computer. For instance, you used a socket to download this article from the web and read it.

The interface for sockets is relatively simple. You start by establishing a connection and then you can read data from the socket or write data into it.

More practically, when you establish a connection with a socket, the operative systems give you back an handle for the connection the handle is called file descriptor. (We will study why it is called a file descriptor, so subscribe to the mail list or follow me on twitter.) After you get the handle, you create a buffer of memory.

To read from a socket, you pass the socket handle and the buffer to the operative system using the read system call. The operative system will first interrupt your process, then it will fill the buffer with data, and then it will resume the execution of your code. Now the buffer contains the data read from the socket and you can act on it.

To write to a socket, you do the inverse operation. You fill your buffer with data. You pass the socket handle and the buffer to the write system call. The operative system interrupts your process, reads the data from the buffer, moves the data to the other end of the socket, and finally resume the execution of your code.

There is much more to study about sockets and I will write about them shortly.

Signal

A less versatile way for processes to get information from the outside world is signal.

Signals are short messages sent to a process, each signal has associated a default action. The default actions are to:

Terminate the process
Ignore the signal
Terminate the process and write possible debug information (dump the core).
Stop (temporarily) the process
Resume the execution of a process stopped earlier.

A signal can be sent from one process to another and can be caught from the process and acted upon.

For instance, several systems catch SIGHUP to reload their configurations. Other systems catch SIGTERM (Ctrl-C) to terminate cleanly the execution and not lose data or important information.

Some signal (SIGKILL and SIGSTOP) cannot be caught, nor ignored, and they just kill the process without giving it the possibility to exit cleanly.

Files

Files are another great way to communicate between processes. One process can write in a file, and another could read from it.

The interface to work with files is exactly the same to work with sockets. You acquire a file descriptor first, then you create a buffer, and finally, invoke the read or write system calls. This similarly would give a hit on why the socket handles are called file descriptor. Underneath they are the same.

Using files for IPC allows processes to communicate between the same machine, they both need to have access to the same file.

Files come also with another complication, there must be coordination between write and read operations. Locks and semaphores are used in these cases.

A great way to use files are IPC is to use tools like SQLite. SQLite can write and read directly from a file and it takes care of all the coordination for you. An application can write data to an SQLite database, while a second one consumes data from it. This approach works great even if it is difficult to receive notifications of writes.

Shared Mapped Memory

The final and most complex way to IPC is mapped memory. This method is not that different from using files but does not rely on an explicit read and write calls. It is quite dangerous because allow data, in the share mapped memory, to be changed implicitly by another process. The data that you read from it, now and in the future, maybe different even if you never changed it.

You can create a shared mapped memory invoking the mmap system call passing the appropriate flags. Other processes can do the same. At this point, processes have access to a memory buffer that is shared between them. What a process writes in the buffer can be read by the others.

Notice how this is the only method that allows implicit changes to the memory owned by the process. With all the other methods you provide a buffer that it is changed when you ask so, using the read system call. With this method, the values stored in the shared memory can change under your feet.

There are more IPC methods but the one provided is an interesting overview, much more than enough to get started.

We continue this chapter on how to work effectively with processes in a Linux system.

Start a process

There are few ways in Linux to create another process. The most used one is the fork system call. It creates a copy of the actual process and starts to run it while the original process keeps running. The fork system call returns a different result on the "parent"/"original" process (it returns 0 zero) from the "child" process (it returns an ID greater than zero). Detecting the return value of the fork call is possible to distinguish between the child and the parent process.

The clone system call is similar but allows for more control. It is a more advanced function and it is used in the implementation of container technologies. It also returns the ID of the new process.

Another related function is execve it does not create a new process, but it executes a new program in place of the original one. It is not strictly related to this topic, but, together with fork is enough to implement a simple shell (like bash or sh or fish).

Process identification

We saw how to start a process with the fork system call, and we see that fork returns an ID. Since the ID identifies a Process, those IDs are called PID, for Process ID.

The number of PID is limited, they are stored in an int, so in systems running for a long time with a lot of processes starting up, the PID will get recycled. However, at any given time, a PID identifies one and only one process.

Knowing the PID of a process is possible to interact with it. Sending it signals or attach it to a debugger.

The simplest way of knowing the PID of a process is to store it when you create it. Of course, it is not always possible since you may need to know the PID of a process not created by you.

Fortunately, there are other ways to know the PID of the processes running in the system at any given moment.

All those methods rely on reading data from /proc. Inside /proc there are several directories, one for each PID, in the form /proc/[PID], listing them is a simple way to know which processes are running in the system at any given moment. To list the process we can rely on bash globing with.

$ ls -lna /proc

This will show a lot of directories. The numeric ones are the PID, while the others are for system information.

Inside those directories, there are a lot of files with very interesting information.

For instance /proc/[pid]/cmdline contains how the software was invoked. /proc/[pid]/exe is a symbolic link to the executable that is running and /proc/[pid]/comm contains the name of the software.

Fortunately, we don’t have to rely on manually parsing the content of the /proc directories since different utilities are available for us.

The ps utility allows to list all the processes running in the system, it is useful to identify the PID of a specific software running in the system. I am writing this article on nvim so if I want to know the PID of nvim I could run:

$ ps -e | grep nvim
27188 pts/1    00:01:08 nvim

And this will show me that the PID of this instance of nvim is 27188.

Another way to use ps is to pass the aux options, this will show more information. Information like the arguments that were passed to the program.

$ ps aux | grep nvim
pgian 27188  3.4  0.1 129616 13764 pts/1    Sl+  17:49   1:29 nvim content/posts/process.asciidoc
pgian 27518  0.0  0.0  14752  1028 pts/2    S+   18:32   0:00 grep --color=auto nvim

We can see the user, the PID, and the command-line invocation of the software. Notice how this shows also grep indeed, grep is matching its own invocations.

While ps is great for quickly get to know the PID of a process, it is not very ergonomic when exploring a running system. A better alternative in such cases is htop. htop allow also to sort the processes in a system to visually show the processes tree.

Here we can my setup as I am writing the article. I am running the software inside tmux (PID 27177), which in turn, runs 3 bash command lines (27731, 27217 and 27178). In turn, one bash shell runs nvim again with PID 27188 and another is running htop (27681).

We started talking about processes, we talk about isolation and why isolation is fundamental, we discover that isolation is great, but we need to do input and output. Then we study some Inter Process Communication (IPC) mechanism. In this last part, we studied how it is possible to see all the processes running in the system with tools like ps and htop.

In this last part, we will understand the state of a process. At the very beginning, we describe processes as a computational unit. Modern computers can run a lot of processes at the same time, while I am writing this, I have 285 processes running. However the number of virtual cores in a computer is limited, my machine has only 4. So how 4 cores can run 285 processes?

Of course, not all processes run at the same time. Some of them sleep.

Process State

It is not necessary for software to keep running continuously. If you are reading or writing from a file descriptor your CPU does not need to work. The CPU is waiting for the IO layer to finish, either against the disk or against the network. In this case, the process is sleeping.

On the contrary, when the CPU is working, for instance, it is summing numbers, creating strings, doing some math operations, moving memory, the process is running.

Besides the sleeping and the running state of a process, there is a third state, the ready state. A process is in the ready state when it has done waiting, for instance, the disk finally finished returned some data, but it is not yet running, because the operative system has not allocated yet resources to it.

Recap

In this chapter, we study processes. We understand what they are and one of their most important features, isolation.

Isolation is great, but software needs to communicate with the outside world, hence several Inter Process Communication (IPC) mechanisms are available. We study briefly, sockets, signals, files, and shared mapped memory.

Then we study how to start a process and how to identify processes in a running machine. We interact with the raw tools that the operative system gives us to query processes and then we upgraded to work with more refined tools like ps and htop.

Finally, we understood what is the state of a process and what are the main states a process can be in, sleeping, running or ready.

If you enjoyed this post, make sure to check out the project jr2sr.com to help you transition in a more senior engineering position.

Also feel free to follow me on twitter: @PauldiGian

Practical DNS

Paul DiGian — Thu, 11 Jun 2020 15:47:00 +0000

DNS

DNS is part of Junior2Senior a course to help you grow as software engineers.

In the first article, we describe what is a web/HTTP request. Requests are directed toward a host and a port.

The port is defined by the protocol, HTTP goes to port 80, HTTPS goes to 443. The host is defined by the request as a string.

A possible host is “google.com” another is “facebook.com”.

However, packets are routed over the internet using IP addresses, numbers, while hosts are strings.

Each IP packet must contain its source and its destination to be delivered. And the host of the source address and of the destination address must be encoded as IP addresses.

In order to create the IP packet and to deliver it to the internet, we need to convert the host name “kitty.com” in its IP address 53.125.12.9.

DNS serves have the basic goal of translating the string that represents the host (“google.com”) in a number, the IP address ( 116.98.109.34) so that a valid IP packet can be generated.

Scratching the surface

At its most basic level, DNS are services that take as input a host string and return the IP address of the host.

A simple utility to test DNS is dig. In its most basic invocation, dig takes an input an host and returns its IPv4.

The following examples have different options to provide a more concise output.

$ dig +noall +answer -t A google.com google.com. 275    IN  A   216.58.205.46 
$ dig +noall +answer -t AAAA google.com google.com. 257 IN  AAAA    2a00:1450:4009:81b::200e

In the first line we translate the host “google.com” into an IPv4 address ( 216.58.205.46).

In the second line, for the same host, we retrieve its IPv6 ( 2a00:1450:4009:81b::200e).

DNS record type

DNS can store different kinds of records beside IP addresses. The most common record type is A that stores IPv4 addresses. The AAAA record stores IPv6 addresses.

Another important record type is the CNAME (or ALIAS) record type. It is useful if a single machine, with only one IP address, runs multiple services on different ports.

The TXT record type can contain arbitrary information, it is usually used to verify the ownership of a domain name.

Other record types are available and widely used, but the working principle is the same for all.

Resolve DNS query

DNS works like any other internet service, but with some peculiarity.

First of all, the IP address of the DNSs are widely know, for instance, the one provided by google is 8.8.8.8 and the one by Cloudflare is 1.1.1.1.

Then DNS works mostly over UDP and not over TCP. The use of UDP is mostly for performance.

Performance is the most strict requirement for DNS.

DNS must be fast.

In order to keep communication as concise and fast as possible, almost all packets DNS queries and responses fit into a single UDP packet and are limited to 512 bytes. If the DNS answer is bigger other communications protocols are used.

Need for speed

The use of UDP over TCP is mostly due to performance requirements. Any application needs to make multiple DNS queries, it must be fast.

Even if fast, it is still impractical to make a DNS query for each network request. Moreover, when visiting a website, most requests are against the same host in a short amount of time.

To address the problem DNS is aggressively cached on different layers.

Cached values eventually goes stale and they must be updated. For this reason, DNS records provide also a TTL value, Time To Live.

After the TTL expires, the DNS value is considered outdated and it must be requested again from the authoritative domain.

Server cache

The first level of DNS cache is the server level. DNS servers don’t store all the values of all the domains, but only a small subset.

If a request for an unknown domain comes, they recursively redirect the request to other DNS until they don’t get the answer from the authoritative (main) DNS. The answer is then cached until the TTL expires.

Client cache

While server cached helps a lot in reducing the latency of DNS queries, it cannot be as fast as a local copy.

Hence the first level of cache regarding the name server happens in the local machine.

The OS intercepts DNS requests and returns a cached answer, as long as the TTL is not expired. As soon as the TTL expires a new request to the server is made and the answer cached again.

TTL and propagation time

Caching the result of DNS queries has the huge benefits of making it fast. The downside is that updates to DNS take time.

All the TTL needs to expire before all the users can see the updated values.

It can take up to 2 days for the information to propagate in the DNS servers. For most applications, the delay is much smaller, a few hours at the very maximum in my experience using AWS Route53 or Cloudfront.

Trivial

DNS can store arbitrary information, and eventually all the DNS servers will agree on a specific values. For these reasons DNS are considered the biggest and more widely deployed eventual consistent databases.

DNS console

After you buy a domain name you can set all the DNS values relative to that domain and all its subdomains.

All the DNS interfaces are similar, offering pretty much the same functionality.

Your DNS provider is mostly a matter of taste and convenience.

AWS Route53

I like to use Amazon Route53 service, that allows buying the domain and set the records in the same interface.

I like Amazon Route53 because:

DNS is a fundamental service for AWS, it will always work
They provide very clear, simple, and cheap pricing
I trust Amazon don’t screw people over a few dollars in the domain registration fee, unlike other vendors.

A domain on AWS cost ~12$ / year plus few cents if your domain is extremely popular or if you need special features.

Beware that other vendors sell your domain at a much higher price BUT they provide a hefty discount for the first year. So it seems like you are buying it for 4$ and the second year they start charging you 50$ / year.

In my opinion, it is just not worth doing business with them.

Trivial

The DNS service of Amazon is called Route53, while Route is quite clear, why not Route 66, much more iconic. Or Route 101? Because 53 is the port used by the DNS protocol.

The console

The console of AWS Route53 looks like this, for each of your domains.

We will focus on the big table. The navigation on the right is for more (advanced) features of AWS Route53.

From the table, we can immediately see the name of the record (first column), the record type ( A, NS, AAAA, CNAME, etc...) and the value of the record.

The highlighted rows are all A records. It is interesting to note how the root domain manage:

The root domain itself, making redbeardlab.com point to the IP address 212.47.232.249
The sub-domain, making simplesql.redbeardlab.com point to 212.47.253.152
All other sub-domain levels, in this case making telemetrics.redisql.redbeardlab.com point to 51.15.142.13

The AAAA records are very similar to the A record.

In this example we make telemetrics.redisql.redbeardlab.com point to 2001:bc8:4400:2c00::2b:c17.

This does not generate an ambiguity with the A record above. If the client asks for the IPv4 address the server returns 51.15.142.13. While, if the client ask for the IPv6 address the server returns 2001:bc8:4400:2c00::2b:c17. In this particular case, both addresses point to the same machine. But they could point to different servers offering completely different services.

The CNAME record creates an alias.

In the first example, we set the host redisql.redbeardlab.com to point to redbeardlab.github.io. In this way we can serve, under the domain we control, content from Github pages, for free. Quite convenient.

In the second case we redirect all the calls to AWS Cloudfront, a CDN managed by AWS.

When a client finds a CNAME it stops asking for the original host IP address and starts asking for the IP address of the ALIAS. If a client goes to redisql.redbeardlab.com, at first it will try to find the IP address of redisql.redbeardlab.com, however, the DNS server will say that the IP address does not exist, but it exists an alias, redbeardlab.github.io. At this point, the client will start looking for the IP address of redbeardlab.github.io.

The TXT fields were used for verifying the ownership of the domain against Google.

This is the most common use I encounter, but other uses for the TXT field are possible.

Finally, the NS record, declare who manage the DNS entry.

In this case, the DNS is managed by… Cloudflare. So all we said above is ignored by a real DNS client that will make its queries not against AWS Route53 but against Cloudflare.

While the TTL (Time To Live) for most records is set to 300 or 600 seconds (5 to 10 minutes) the TTL for the NS is set to a much larger value, 7200 seconds (2 hours).

This is to be expected, you don’t change often the NS value, once you set your DNS provider you tend to stick to it. Moreover, change the NS value is expensive, all the queries need to be directed to a completely different service that needs time to accommodate it.

Note on Name Server configuration in AWS Route53

Unfortunately, in Route53, it is not sufficient to set the Name Server record set, but it is also necessary to set the name servers in another setting.

Under 'Register Domains' > $your_domain and then "Add or edit name servers".

I find a good practice to keep the two values in sync.

Cloudflare

Another very reputable DNS provider is Cloudflare. They don’t sell you domain names directly, but they can manage them for you.

So you will need to buy your domain somewhere else, also an AWS Route53, and then let Cloudflare manage it. This is what I did above.

This is done setting up the authoritative name server to point to Cloudflare. It is as simple as adding a new record (of type NS) to your existing DNS records (modulo the extra setting in AWS Route53). From that moment on, beside propagation delay, all the DNS queries will be redirected toward the new authoritative name server.

The console

The console of Cloudflare is different from the one of Route53, but overall they contain the same information and it is equally simple to use.

We can recognize the same kind of information:

the A fields that point to IPv4 addresses
the AAAA fields that point to IPv6 addresses
the CNAME used for alias
the TXT again used for domain ownership verification
the MX used for email routing, with a priority

The “Proxy Status” column in the Cloudflare console indicates whenever Cloudflare takes care only of the DNS ( DNS only) or also to cache and manage with their services the request to the domain.

Recap

Hopefully, this article is helpful if you are setting up your DNS or if you are getting started with web services.

If you have any question I would love to expand the article and help you out, feel free to reach over on twitter @pauldigian.

Originally published at https://jr2sr.com on June 7, 2020.

Always check your resources before to free them

Paul DiGian — Wed, 29 Apr 2020 20:25:18 +0000

Today I fixed a minor bug in our production code.

The software is written in Go(lang) and makes extensive use of the defer statement.

The defer statement, take a function invocation and run it after the current function returns.

They are very handy for clean-up tasks. A classical example is the acquisition of a lock. Or closing a file.

Without defer you would get your resource at the beginning of the function, use it, and then clean-up.

In the case of locks, you would acquire the lock, do the work in the critical section, release the lock, and the return.

In case of files, you would open the file, write and read to the file, and then close the file.

With this approach, every return path must be checked. As soon as you add a new return path, you must make sure that it closes all the resources it may have acquired. This is simpler to say than to do.

Code bases grow larger and larger and it gets messy to check all the possible returns path. And it also looks ugly and not DRY.

defer solve this problem. As soon as you get the resource you defer its cleanup. As soon as you acquire the lock, you defer the release of it. As soon as you open a file, you defer closing it.

You want to put the defer as close as possible to the resource initialization for two reasons.
First of all, it is clearer when reading the code.
Second, the closer the two calls are, the less it is likely that some hidden return path sneaks into the acquisition and the cleanup.

But you don't want to put it too close. Before invoking the defer you must make sure that the resource is valid.

Today bug was about an HTTP call. I was making the HTTP call and immediately after I was closing its body to avoid leaking resources.

client := &http.Client{}    
resp, err := client.Do(req)
defer resp.Body.Close()      
if resp.StatusCode >= 400 {                             
    err = fmt.Errorf("Some error happened %s", resp.Status)
    return
}

Unfortunately, sometimes, the network fails. And today it failed.

The network failed, the StatusCode was greater than 400 and the functions returned early. As soon as the function returned, it invoked the defer statement: resp.Body.Close().

Since the requested had failed, the Body was not there and there was a nil pointer.

The defer ended up calling the Close() method of nil and this caused a runtime panic, blowing it up in production.

I corrected the code with:

client := &http.Client{}                                                                                                                                                          
resp, err := client.Do(req)                                                                                                                              
if err != nil { 
        err = fmt.Errorf("The HTTP request failed %s", err)                                                         
        return 
} 
defer resp.Body.Close()

First, I check that the request succeed. If the request succeed, the Body is something we can Close(), and so we can defer its closing.

If the request failed, for whatever reason, we return early, without deferring the close of the Body, that does not exist.