DEV Community: Nicolas Mesa

Booting Your Own Kernel on Raspberry Pi via Uart

Nicolas Mesa — Wed, 21 Aug 2019 02:13:02 +0000

Cross-posted from my personal blog https://blog.nicolasmesa.co.

For the last few weeks, I’ve been reading a great book about building your own operating system for a Raspberry Pi from scratch. It has been a great experience, and I’ve learned a lot. One thing I didn’t like was my development workflow, which looked like this:

Make a code change.
Build the change.
Unplug the Raspberry Pi.
Remove the SD card.
Plug the SD card to my laptop.
Copy the newly built kernel over to the SD card.
Eject the SD card.
Place the SD card back on the Raspberry Pi.
Connect the Raspberry Pi to my computer.
Close previous terminal since the screen command made it pretty useless.
Connect to the Raspberry Pi using the screen command.
Test my change.

All of this was pretty time consuming making the feedback loop very slow (especially, when I forgot a crucial step such as placing the SD card back in the Raspberry Pi). I searched to see if there was a way to improve my workflow and found an issue in the github repo called UART boot. I read through it and decided to build the UART boot myself as a learning exercise. In this blog post, I go through the process that I took to make UART boot work, including my wrong assumptions/failures and how I overcame them.

Requirements

To make this project worth my time, I decided that it needed to be able to do the following:

The kernel must act as both the UART bootloader and the kernel itself (a single executable).
I must be able to send a newly compiled kernel over UART, and the Raspberry Pi must boot.
I must be able to have an interactive session over UART using my program (basically, replace screen).
The screen replacement must leave my terminal in a working state (I assume there’s a way to make this work in screen but, I wanted to do this as a learning exercise).

Prerequisites

If you want to follow along, you’ll need:

To meet all the prerequisites from the book.
To have completed the first lesson.
To have completed the second exercise to use UART instead of Mini UART (this might still work with Mini UART, but I haven’t tried).
Python 3.7 (This is the version I used).

Note If you haven’t completed the exercise and would still like to follow along, follow the setup instructions below. I also tagged each commit to make it easier to follow along/jump ahead.

Setup

If you already have your project, make sure that you’ve built your kernel and have it in the SD card. Otherwise, you can clone my repository and can follow along:

$ git clone https://github.com/nicolasmesa/PiOS
$ git checkout tags/uart-boot-initial

Git tags

I’ve created commit tags for each section in case you want to experiment with a specific part of the project. You can checkout the tags by running the command:

$ git checkout tags/<tag-name>

I’ll reference the tag name at the end of each section.

Git tag: uart-boot-initial - link

`screen` replacement

To be able to send the kernel over UART, we need to establish a serial connection with the Raspberry Pi and send the kernel bytes. The screen replacement will do the same thing with the difference that it will also read from the serial connection and stdin and will write to stdout. If we’re able to make this work, sending the kernel should be easy.

Let’s get started by installing pyserial (I suggest you do this in a virtual environment):

(rpi_os) $ pip install pyserial==3.4

Let’s create a file called boot_send.py and add the following code:

import os
import select
import serial
import sys
import time
import tty

class UartConnection:

    def __init__ (self, file_path, baud_rate):
        self.serial = serial.Serial(file_path, baud_rate)

    def send_string(self, string):
        return self.send_bytes(bytes(string, "ascii"))

    def send_bytes(self, bytes_to_send):
        return self.serial.write(bytes_to_send)

    def read(self, max_len):
        return self.serial.read(max_len)

    def read_buffer(self):
        return self.read(self.serial.in_waiting)

    def read_buffer_string(self):
        return self._decode_bytes(self.read_buffer())

    def start_interactive(self, input_file, output_file):
        try:
            tty.setcbreak(input_file.fileno())
            while True:
                rfd, _, _ = select.select([self.serial, input_file], [], [])

                if self.serial in rfd:
                    r = self.read_buffer_string()
                    output_file.write(r)
                    output_file.flush()

                if input_file in rfd:
                    r = input_file.read(1)
                    self.send_string(r)
        except KeyboardInterrupt:
            print("Got keyboard interrupt. Terminating...")
        except OSError as e:
            print("Got OSError. Terminating...")
        finally:
            os.system("stty sane")

    def _decode_bytes(self, bytes_to_decode):
        return bytes_to_decode.decode("ascii")

Wow, that’s a lot of code! Let’s go through it.

Constructor

The constructor (__init__) receives two arguments which we use to create a serial connection:

file_path: The path to the file where the UART device is (/dev/cu.SLAB_USBtoUART in my computer).
baud_rate: The baud rate to use in the serial connection.

Utility methods

We have a bunch of utility methods with varying levels of abstraction, but they all end up calling either send_bytes (which calls serial.write) or read (which calls serial.read).

start_interactive

The start_interactive method receives two arguments (input_file and output_file). We will use these arguments as stdin and stdout, respectively. The first thing this method does is call tty.setcbreak(input_file.fileno()) on the stdin file. tty.setcbreak turns the tty into “raw” mode and allows us to read the characters as they are typed instead of buffering them until the user presses enter.

The select function receives a set of file descriptors to monitor. When called, it blocks until one of those files has data that we can read. When the select function returns, the rfd variable has a list of file descriptors that have data available. Next, we verify if the serial file descriptor has data available, and, if it does, we read it all and send the output to our terminal (output_file). We also check if our input_file has data, and, if it does, we send it over the serial connection to the Raspberry Pi. Note that in this case, we don’t write anything to the output_file and we rely on the Raspberry Pi echoing back the character that we sent.

The os.system("stty sane") call makes our tty not be in raw mode anymore. This function allows us to keep using the same terminal (instead of the problem I had with screen).

def main():
    import sys
    uart_connection = UartConnection(
        # Change these to match your setup
        file_path='/dev/cu.SLAB_USBtoUART',
        baud_rate=115200
    )
    time.sleep(1)
    uart_connection.start_interactive(sys.stdin, sys.stdout)

if __name__ == ' __main__':
    main()

Now, let’s execute the script. Assuming everything worked, you should see something like this:

(rpi_os) $ python boot_send.py
Hello world!

You should be able to type anything you want, and the Raspberry Pi should echo it back. Press ctrl-c to exit, and your terminal should still work normally.

Note: If you’re using the git branch I provided, you’ll need to hit enter to see the “Hello world!” message. I did this to make sure that the kernel only sends the Hello world! message after boot_send has connected successfully.

Git tag: uart-boot-screen-replacement - link

Sending the kernel over UART

Sending the kernel should be no different than sending characters over serial; everything gets translated to bytes, after all.

Protocol

We’ll use the following protocol to send the kernel:

The Raspberry Pi boots into the kernel and blocks while waiting to read a single line.
If the line it reads matches the word kernel then we go into UART booting mode, if not, we skip the UART boot.
When in UART booting mode, the boot_send script sends an integer (4 bytes) containing the size of the kernel that we’re about to send.
The Raspberry Pi takes note of this size and sends it back.
The boot_send script verifies that the Raspberry Pi got the correct number.
The boot_send script starts reading the kernel and sends it byte by byte over the serial connection.
The Raspberry Pi receives each byte and…
1. Keeps a running sum of all the bytes (a checksum).
2. Places that byte in memory (overwriting the current kernel).
When the Raspberry Pi receives the number of bytes equal to the size of the kernel (sent in a previous step), it sends the checksum over UART.
The boot_send script verifies that the checksums match (for error detection purposes).
The Raspberry Pi responds with the string "Done" to denote that it will now jump to the new kernel.
The Raspberry Pi jumps back to address 0x00 to start executing in the new kernel.
The boot_send script starts an interactive session.

One thing that we need to keep in mind is endianness when sending the size. I decided to use big-endian for all communication.

boot_send modifications

We need to add some functionality to our boot_send script to implement the protocol:

UartConnection

Let’s add these functions to our UartConnection class:

class UartConnection:
    # ...
    def send_line(self, line):
        if not line.endswith("\n"):
            line += "\n"
        return self.send_string(line)

    def send_int(self, number):
        if number > 2 ** 32 - 1:
            raise 'Number can only be 4 bytes long'
        number_in_bytes = number.to_bytes(4, byteorder='big')
        return self.send_bytes(number_in_bytes)

    def read_int(self):
        bytes_to_read = 4
        number_bytes = self.read(bytes_to_read)
        return int.from_bytes(number_bytes, byteorder='big')

    def read_line(self):
        return self._decode_bytes(self.serial.readline())

The most important thing to call out here is the byteorder which we decided to use (big stands for big-endian).

Kernel checksum

We compute a checksum to validate against some errors. I don’t think this is the best way to check for errors since something as simple as changing the order of two bytes would still make this checksum pass. I use it because it gives me a bit more confidence that things are working as expected.

def compute_kernel_checksum(kernel_bytes):
    num = 0
    for b in kernel_bytes:
        num = (num + b) % (2 ** 32)
    return num

We wrap around at 2^32 because that’s the max size of an integer.

send_kernel

Let’s add a function to send the kernel to the Raspberry Pi following our protocol:

def send_kernel(path, uart_connection):
    with open(path, mode='rb') as f:
        uart_connection.send_line("kernel")
        kernel = f.read()
        size = len(kernel)
        checksum = compute_kernel_checksum(kernel)

        print("Sending kernel with size", size, "and checksum", checksum)
        uart_connection.send_int(size)
        time.sleep(1)
        size_confirmation = uart_connection.read_int()
        if size_confirmation != size:
            print("Expected size to be", size, "but got", size_confirmation)
            return False

        print("Kernel size confirmed. Sending kernel")
        uart_connection.send_bytes(kernel)
        time.sleep(1)

        print("Validating checksum...")
        checksum_confirmation = uart_connection.read_int()
        if checksum_confirmation != checksum:
            print("Expected checksum to be", checksum,
                  "but was", checksum_confirmation)
            return False

        line = uart_connection.read_line()
        if not line.startswith("Done"):
            print("Didn't get confirmation for the kernel. Got", line)
            return False

        return True

Our function receives a path to the kernel file and a UartConnection object. We use these to follow the protocol described above to send the kernel to the Raspberry Pi.

Calling the `send_kernel` function

Let’s make a small tweak to our main function.

def main():
    # ...
    uart_connection = UartConnection(...)
    time.sleep(1)
    result = send_kernel(
        path="kernel8.img",
        uart_connection=uart_connection
    )
    if result:
        print("Done!")
        uart_connection.start_interactive(sys.stdin, sys.stdout)
    else:
        print("Error sending kernel :(")

Our script sends the kernel first and then starts an interactive session.

Git tag: uart-boot-boot-send-client-side - link

Kernel side modifications

Let’s dive into the modifications needed for the kernel.

branch_to_address

The last part of our boot protocol involves jumping back to address 0x00 and starting the execution from that memory location. Let’s start by adding a function that can do that.

Open the utils.h file and add the function declaration:

extern void branch_to_address( void * );

Now, go to the utils.S file and add the definition:

global branch_to_address
branch_to_address:
    br x0

uart_send_int / uart_read_int

We’ll need to send and read integers via UART (for the kernel size and the checksum). Let’s open the uart.h file and add the following declarations:

void uart_send_int(int number);
int uart_read_int();

Let’s implement the functions in the uart.c file:

int uart_read_int() {
    int num = 0;
    for (int i = 0; i < 4; i++) {
        char c = uart_recv();
        num = num << 8;
        num += (int)c;
    }
    return num;
}

void uart_send_int(int number) {
    uart_send((char)((number >> 24) & 0xFF));
    uart_send((char)((number >> 16) & 0xFF));
    uart_send((char)((number >> 8) & 0xFF));
    uart_send((char)(number & 0xFF));
}

In the uart_read_int, we read one byte at a time (most significant byte first) and keep shifting those bytes to the left until we get the 4 bytes that represent the integer. In the uart_send_int, we send one byte at a time (most significant byte first). Note that using pointers here is not a good idea since we shouldn’t make assumptions on how the Raspberry Pi stores ints in memory (big-endian vs. little-endian).

include utils.h

Open the kernel.c file and add the following #include statement:

#include "utils.h"

readline

Here we write a pretty simple readline function:

int readline(char *buf, int maxlen) {
    int num = 0;
    while (num < maxlen - 1) {
        char c = uart_recv();
        if (c == '\n' || c == '\0' || c == '\r') {
            break;
        }
        buf[num] = c;
        num++;
    }
    buf[num] = '\0';
    return num;
}

strcmp

Our protocol needs to read a line of text and compare it against the word kernel. To do that, we need a string comparison function:

int strcmp(char *str1, char *str2) {
    while (1) {
        if (*str1 != *str2) {
            return *str1 - *str2;
        }

        if (*str1 == '\0') {
            return 0;
        }

        str1++;
        str2++;
    }
}

copy_and_jump_to_kernel

Now we have all the pieces to implement the protocol we brought up before:

void copy_and_jump_to_kernel() {
    int kernel_size = uart_read_int();
    uart_send_int(kernel_size);

    char *kernel = (char *)0;
    int checksum = 0;
    for (int i = 0; i < kernel_size; i++) {
        char c = uart_recv();
        checksum += c;
        kernel[i] = c;
    }
    uart_send_int(checksum);

    uart_send_string("Done copying kernel\r\n");
    branch_to_address((void *)0x00);
}

This function receives the kernel size and sends it back. Then, receives each byte from the new kernel and places it in memory starting on address 0x00. Next, it sends the calculated checksum and a string saying "Done copying the kernel". Finally, it branches to start executing at address 0x00 (where the new kernel resides).

kernel_main

Let’s add the logic to read a line on boot, and compare it against kernel to decide if we want to boot over UART or not:

void kernel_main(void) {
    int buff_size = 100;
    uart_init();

    char buffer[buff_size];
    readline(buffer, buff_size);
    if (strcmp(buffer, "kernel") == 0) {
        copy_and_jump_to_kernel();
    }

    uart_send_string("Hello world!\r\n");
    while (1) {
        uart_send(uart_recv());
    }
}

Note that the call to copy_and_jump_to_kernel never returns.

Git tag: uart-boot-boot-send-kernel-side - link

Testing our UART boot

Build the kernel and copy it to the SD card

Here are the steps I took to build the kernel and copy it to the SD card on my Mac:

$ ./build.sh
$ cp kernel8.img /Volumes/boot/

Booting test: sending the same kernel

Now eject the SD card and put in your Raspberry Pi.

Let’s run our boot_send script:

(rpi_os) $ python boot_send.py 
Sending kernel with size 1991 and checksum 201267
Kernel size confirmed. Sending kernel
Validating checksum...
Done!

If you see this, it means that you just sent a new kernel (the same kernel that you installed in the SD card) over UART! At this point, the kernel is stuck in the readline function to check whether you want to send a new kernel or not. Let’s press enter:

Hello world!
Got keyboard interrupt. Terminating...

Press ctrl-c to exit.

Booting test: making a small change

Sending the same kernel is not why we did all this work! Let’s make a small change to our kernel code, compile it and send it over UART (instead of putting it in the SD card). Go to the kernel_main function and change the string that we send:

int kernel_main(void) {
    // ...
   uart_send_string("Hello from a new kernel!!!\r\n"); 
   // ...
}

Let’s compile it and execute our python script (no need to copy anything to the SD card anymore!):

(rpi_os) $ ./build.sh
(rpi_os) $ python boot_send.py 
Sending kernel with size 2005 and checksum 202127
Kernel size confirmed. Sending kernel
Validating checksum...
Done!

Great! We sent the kernel! Now, the moment of truth! Let’s press enter:

Hello from a new kernel!!!
Got keyboard interrupt. Terminating...

Git tag: uart-boot-string-change-test-uart-boot - link

Booting test: adding a function

We did it! We’re done, right? Well, not so fast! Let’s try to make a different change. Let’s add a function and call it from kernel_main. Open the kernel.c file and add the following code:

// ...
void my_test_function(void) {
    uart_send_string("Sending a test message!\r\n"); 
}

void kernel_main(void) {
    // ...
   uart_send_string("Hello from a new kernel!!!\r\n");  
   my_test_function();
   // ...
}

We added a new function called my_test_function, and we call it from kernel_main. Let’s compile and send this over UART:

(rpi_os) $ ./build.sh
(rpi_os) $ python boot_send.py Sending kernel with size 2077 and checksum 208704
Kernel size confirmed. Sending kernel
Validating checksum...

It hangs in the Validating checksum… part! So, what’s going on? Let’s think for a minute about what we’re doing. The Raspberry Pi loads our kernel from the SD card onto address 0x00 and starts executing code in that address. When we copy the kernel over (over UART), we overwrite the kernel from the SD card with the new kernel. If the kernels are the same, it won’t make a difference since we’re effectively leaving the code as-is. Our string change didn’t affect us either because the strings go in the .rodata (read-only data) section of the executable which goes after the .text section (where the executable code lives), so changing the string doesn’t modify the addresses of the executable code. Adding a function, however, changes the .text layout causing the kernel to “corrupt” itself while it copies the new kernel.

Git tag: uart-boot-fail-by-adding-function - link

The fix

Before implementing a fix, let’s remove my_test_function to get back into a “good” state.

Git tag: uart-boot-remove-test-function - link

Options

So, how do we fix this problem? Let’s consider these two options:

Option 1: Copy the kernel to a different location and jump there

Using this option would not overwrite the kernel that is currently running, and we would jump to the new kernel in a different address. There’s a few drawbacks to this approach:

The kernel code will not start in address 0x00. The course might make some assumptions about the kernel being in address 0x00, and we would be breaking those assumptions.
The course could use the address range that we choose to put our new kernel.
We wouldn’t be able to send a kernel multiple times over UART in the same session (once we copy over the kernel and we’re running in the new address, we wouldn’t have a new place to copy another new kernel).

Option 2: Copy the current running kernel to a new address range, jump to it, copy the new kernel to the original address range, jump back to it

To implement this option, we need to:

Copy the currently running kernel to a new memory location (let’s say 0x8000).
Jump to a function in the new address range.
The function in the new memory location implements the protocol described above (our copy_and_jump_to_kernel function).
- It copies the kernel over UART starting at memory location 0x00.
Jump back to address 0x00.
Start running the new kernel.

We treat the address range starting at 0x8000 as temporary storage that we can use for other purposes once we jump back to address 0x00.

With this implementation, we’re able to keep all the assumptions that the course might be making and we’re able to copy the kernel multiple times over the same session! This approach is called chain loading. You can read more about chain loading here.

Fix implementation

Position Independent Code

Currently, the compiler compiles our kernel with the assumption that its code starts at address 0x00. The compiler can then hardcode the absolute addresses of every function using this. Absolute addresses become a problem once we start executing instructions in the kernel that starts at location 0x8000 because it would still be referencing the old absolute memory locations.

We can solve this issue by using Position Independent Code, where the addresses are relative to the address held by the program counter.

To enable this, we need to tweak our Makefile to add the -fPIC flag. You can read more about this flag in this Stack Overflow post.

# -fPIC makes the addresses relative instead of absolute allowing
# us to place the kernel anywhere in memory.
COPS = -fPIC -Wall -nostdlib -nostartfiles -ffreestanding -Iinclude -mgeneral-regs-only
ASMOPS = -fPIC -Iinclude

This change is all that is needed to make our code position independent!

Determining current kernel size

The kernel needs to know its size to be able to copy itself. We use the bss_end section of the linker.ld file to determine the size of the kernel. Let’s add a reference to that section in our kernel.c file:

#include "utils.h"
// ...
extern char bss_end[];

We define this as an array, even though it technically isn’t (you can read more about why in this post). Now we can use bss_end as a pointer to the address where the kernel ends (this includes the .bss).

copy_current_kernel_and_jump

Let’s implement a function that:

Copies the current kernel (starting in address 0x00) to the new address range (starting at 0x8000).
Jumps to the copy_and_jump_to_kernel function, that we wrote before, in the new address range.

Note: Make sure to define this function below the copy_and_jump_to_kernel.

void copy_current_kernel_and_jump(char *new_address) {
    char *kernel = (char *)0x00;
    char *end = bss_end;

    char *copy = new_address;

    while (kernel <= end) {
        *copy = *kernel;
        kernel++;
        copy++;
    }

    // Cast the function pointer to char* to deal with bytes.
    char *original_function_address = (char *)&copy_and_jump_to_kernel;

    // Add the new address (we're assuming that the original kernel resides in
    // address 0). copied_function_address should now contain the address of the
    // original function but in the new location.
    char *copied_function_address =
        original_function_address + (long)new_address;

    // Cast the address back to a function and call it.
    void (*call_function)() = (void (*)())copied_function_address;
    call_function();
}

This function copies the currently running kernel to the address range starting at new_address. Then, it does some pointer math to determine the address of the copy_and_jump_to_kernel function in the new address range. Finally, it calls that function in the new address range.

Call `copy_current_kernel_and_jump` from `kernel_main`

All that remains is to call the copy_current_kernel_and_jump from kernel_main and, we should be ready to test:

void kernel_main(void) {
    // ...
    if (strcmp(buffer, "kernel") == 0) {
        copy_current_kernel_and_jump((char *)0x8000);
    }
    // ...
}

Git tag: uart-boot-fix-implementation - link

Testing the fix

First, we need to build our kernel and copy it to the SD card. Refer to the Build the kernel and copy it to the SD card section.

Let’s perform the same tests we ran before to make sure that we can still send the kernel over UART. Refer back to the following sections:

Booting test: sending the same kernel: This will help us make sure that the boot process works end-to-end.
Booting test: making a small change: This test will help us make sure that everything is working at least as well as it was working before the fix.

Testing the previous error

Now, the moment of truth! Let’s make a change that adds a function just like in the Adding a function section. Let’s test it out!

(rpi_os) $ ./build.sh
(rpi_os) $ python boot_send.py 
Sending kernel with size 2280 and checksum 223784
Kernel size confirmed. Sending kernel
Validating checksum...
Done!

This output is promising! At least our boot_send script didn’t hang in the Validating checksum… step like it did last time. The new kernel probably booted up and is stuck in the readline function. Let’s press enter to send an empty line:

Hello from a new kernel!!!
Sending a test message!
Got keyboard interrupt. Terminating...

We did it! We were able to add a new function to the kernel, and it was still able to boot up!

Sending the same kernel multiple times

Now, what if we try to send the same kernel over and over again? We would be able to send a new kernel that handles the next UART boot differently without having to put it in the SD card!

(rpi_os) $ python boot_send.py 
Sending kernel with size 2440 and checksum 234324
Kernel size confirmed. Sending kernel
Validating checksum...
Done!

Nothing surprising here, our kernel is stuck in the readline function again. This time, instead of pressing enter, let’s press ctrl-c to exit out of our boot_send program and rerun it.

Got keyboard interrupt. Terminating...
(rpi_os) $ python test.py 
Sending kernel with size 2440 and checksum 234324
Kernel size confirmed. Sending kernel
Validating checksum...
Done!

We sent the kernel again in the same session! Let’s press enter to make sure that our new kernel booted:

Hello from a new kernel!!!
Sending a test message!
Got keyboard interrupt. Terminating...

Git tag: uart-boot-succeeds-when-adding-function - link

Conclusion

In this post, we built a kernel that can be booted via UART to test, or from the SD card (for more permanent solutions). To do this, we came up with a protocol for the UART communication and implemented it. After a few tests, we noticed there was a problem when we tried to boot a kernel with an extra function. The problem was that the kernel was overwriting/corrupting its instructions. To overcome this problem, we made the kernel first copy itself to a new address range and perform the UART copy while executing from that address range. After the kernel finishes copying the new kernel over UART, it can jump back to the original start address (0x00) and execute the new kernel.

Improvements

Here is a small list of improvements that you can make to the project:

Add arguments to the boot_send script. I did this in my project, but this post is already long as it is.
Clean the kernel.c file and extract the functions to a helper file.
Use the C preprocessor to exclude the UART boot section if you don’t pass a UARTBoot flag.
Make UART boot work with multiple CPUs. I implemented this as well in a somewhat hacky way (I might write a post about it later).
Add a better checksum (CRC, for example).

Links / References

SaaS Like Isolation in Django Rest Framework

Nicolas Mesa — Mon, 08 Oct 2018 04:15:59 +0000

Cross-posted from my personal blog https://blog.nicolasmesa.co.

Hi there! In this post, we talk about how to add SaaS-like isolation to a Django Rest Framework service without bringing in third-party libraries.

First, let’s start with why? I wanted to learn how to use Django Rest Framework by building a small SaaS service. I started googling to see if there were any resources available. Most of them suggested to use a third-party library such as Django Organizations, Django tenant schemas or use DB namespaces, all of which are only compatible with Postgres. I didn’t want to rely on a third-party library for this, since this would mean, significant ramp-up time, another thing that could break, another thing to update, and another layer of abstraction that could end up getting in my way. I decided to do this myself.

If you want to follow along, here’s what you will need:

A computer with python 3 installed (I used python 3.6).
A virtual environment already created and with Django (I used version 2.1.2) and Django rest framework (I used version 3.8.2).

We use SQLite for the database since it doesn’t make a difference for this example. Note that if you’re planning on using this for production, you should not use SQLite.

Use-cases

For our sample project, we’re going to build the backend of a messaging API for a company. Here is a list of use-cases that our app needs to support:

Users must be able to…

signup. A Company should be created automatically.
create other users in the same company (we’re going to ignore admin users and permissions).
see a list of users from the same company.
send a message to another user in the same company.
see the messages they sent or received.

Users must NOT be able to…

see the users from other companies.
read messages from other companies.
send messages to users in other companies.
see messages from other users (even in the same company).

Start the project

Let’s start our project. I name mine saas_django.

$ django-admin startproject saas_django
$ cd saas_django
$ tree | grep -v pyc
.
├── manage.py
└── saas_django
    ├── __init__.py
    ├── settings.py
    ├── urls.py
    └── wsgi.py

1 directory, 5 files

Note: I usually commit all my changes every time I start a project or create a new app. This is an excellent way to provide checkpoints for yourself to be able to go back if needed. I’m going to skip the git commands in here, but I encourage you to use version control to track your project.

The accounts app

Let’s create an app called accounts.

$ python manage.py startapp accounts
$ tree | grep -v pyc
.
├── accounts
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── migrations
│   │   └── __init__.py
│   ├── models.py
│   ├── tests.py
│   └── views.py
├── manage.py
└── saas_django
    ├── __init__.py
    ├── settings.py
    ├── urls.py
    └── wsgi.py

4 directories, 14 files

Let’s open models.py in the accounts folder and put in the following code.

import uuid
from django.contrib.auth.models import AbstractUser
from django.db import models, transaction

class CompanyManager(models.Manager):
    """Manager for the Company model. Also handles the account creation"""

    @transaction.atomic
    def create_account(self, company_name, username, password, company_address=None):
        """Creates a Company along with the User and returns them both"""

        company = Company(
            name=company_name,
            address=company_address,
        )
        company.save()

        user = User.objects.create_user(
            username=username,
            password=password,
            company=company,
        )

        return company, user

class Company(models.Model):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    name = models.CharField('name', max_length=100)
    address = models.CharField('address', max_length=250, blank=True)

    objects = CompanyManager()

    class Meta:
        db_table = 'companies'

    def __str__ (self):
        return self.name

class User(AbstractUser):
    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    company = models.ForeignKey(Company, related_name='%(class)s', on_delete=models.CASCADE, editable=False)

    class Meta:
        db_table = 'users'

    def __str__ (self):
        return f'({self.company.name}) - {self.username}'

This code defines a CompanyManager, a Company, and a User. Let’s dive into what each of these does:

Company:
- This is the model that represents the SaaS account.
- Every other model should have a ForeignKey that links back to this one.
CompanyManager:
- Provides a create_account method that ensures that a Company is always created with a User assigned to it.
- Tom Christie recommends that we do this to be able to reason where changes to the DB are taking place (see this great video).
User:
- Custom User model. It is recommended not to use Django’s provided User model and create a custom one instead. Changing it afterward seems to be very painful.
- Defines the __str__ method to return the company name inside parentheses and the username. This will be useful later to be able to identify the company that a user belongs to.

Note : I also added the db_table Meta field. This isn’t required, and Django will automatically name the DB table for you (for example the Company table would have the name accounts_company). I decided to add the db_table to follow the advice from this blog post to manually name all of my database tables.

Let’s open up our settings.py and add the following:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    # Add these two lines
    'rest_framework',
    'accounts',
]

AUTH_USER_MODEL = 'accounts.User'

We add rest_framework and our accounts app to the INSTALLED_APPS list. We also override the default User model and use our own instead by setting AUTH_USER_MODEL to accounts.User.

Let’s create our migrations and migrate our database.

$ python manage.py makemigrations
Migrations for 'accounts':
  accounts/migrations/0001_initial.py
    - Create model User
    - Create model Company
    - Add field company to user
    - Add field groups to user
    - Add field user_permissions to user
$ python manage.py migrate
Operations to perform:
  Apply all migrations: accounts, admin, auth, contenttypes, sessions
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0001_initial... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_log in_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying accounts.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying sessions.0001_initial... OK

Our database is now created along with the users and companies tables. Let’s start working on our serializers. Create a new file called serializers.py inside of the accounts folder and add the following code:

from django.contrib.auth import get_user_model
from rest_framework import serializers
from .models import Company

User = get_user_model()

class UserSerializer(serializers.HyperlinkedModelSerializer):

    class Meta:
        model = User
        fields = (
            'url',
            'id',
            'username',
            'password',
        )
        # Make sure that the password field is never sent back to the client.
        extra_kwargs = {
            'password': {'write_only': True},
        }

    def create(self, validated_data):
        return User.objects.create_user(**validated_data)

    def update(self, instance, validated_data):
        updated = super().update(instance, validated_data)

        # We save again the user if the password was specified to make sure it's properly hashed.
        if 'password' in validated_data:
            updated.set_password(validated_data['password'])
            updated.save()
        return updated

class CompanySerializer(serializers.HyperlinkedModelSerializer):

    class Meta:
        model = Company
        fields = (
            'id',
            'name',
            'address',
        )

class AccountSerializer(serializers.Serializer):
    """Serializer that has two nested Serializers: company and user"""

    company = CompanySerializer()
    user = UserSerializer()

    def create(self, validated_data):
        company_data = validated_data['company']
        user_data = validated_data['user']

        # Call our CompanyManager method to create the Company and the User
        company, user = Company.objects.create_account(
            company_name=company_data.get('name'),
            company_address=company_data.get('address'),
            username=user_data.get('username'),
            password=user_data.get('password'),
        )

        return {'company': company, 'user': user}

    def update(self, instance, validated_data):
        raise NotImplementedError('Cannot call update() on an account')

We create 3 serializers:

UserSerializer
- Serializes the User model.
- Makes the password field write-only to prevent our clients from ever seeing a password (even if it’s hashed).
- Overrides the update method to properly handle a password update (not the most elegant solution since it involves two writes to the DB but it’s the only way I found that reuses the parent’s logic).
CompanySerializer
- Serializes the Company model.
AccountsSerializer
- Uses the previously defined serializers.
- Calls the create_account method that we added in CompanyManager to create a new Company and User.
- Overrides the update method to throw an exception if called. We do this because we only want this serializer to be used for account creation. After that, an edit can be made directly in the /api/v1/users endpoint or in the /api/v1/company endpoint (not implemented yet).

Let’s create the views for our accounts, users, and companies. Open the views.py file in the accounts directory and add the following code:

from django.contrib.auth import get_user_model
from rest_framework import generics, permissions
from . import serializers

User = get_user_model()

class AccountCreate(generics.CreateAPIView):
    name = 'account-create'
    serializer_class = serializers.AccountSerializer

class UserList(generics.ListCreateAPIView):
    name = 'user-list'
    permission_classes = (
        permissions.IsAuthenticated,
    )
    serializer_class = serializers.UserSerializer
    queryset = User.objects.all()

    def perform_create(self, serializer):
        company_id = self.request.user.company_id
        serializer.save(company_id=company_id)

    def get_queryset(self):
        # Ensure that the users belong to the company of the user that is making the request
        company_id = self.request.user.company_id
        return super().get_queryset().filter(company_id=company_id)

class UserDetail(generics.RetrieveUpdateDestroyAPIView):
    name = 'user-detail'
    permission_classes = (
       permissions.IsAuthenticated,
    )
    serializer_class = serializers.UserSerializer
    queryset = User.objects.all()

    def get_queryset(self):
        # Ensure that the user belongs to the company of the user that is making the request
        # Note that this method is identical to the one in `UserList`
        company_id = self.request.user.company_id
        return super().get_queryset().filter(company_id=company_id)

class CompanyDetail(generics.RetrieveUpdateAPIView):
    name = 'company-detail'
    permission_classes = (
        permissions.IsAuthenticated,
    )
    serializer_class = serializers.CompanySerializer

    def get_object(self):
        # Ensure that users can only see the company that they belong to
        return self.request.user.company

We define 4 views:

AccountCreate
- Only meant for account creation purposes.
- Inherits from CreateAPIView so there won’t be a way to list or update the accounts.
UserList
- Provides a list of users.
- Overrides the get_queryset method to return only the results related to the company of the caller.
UserDetail
- Provides a view for a single user.
- Overrides the get_queryset method in the same way the UserList view does (we’ll address this later). If you try to get a user from a different company, you will see a 404 Not Found response.
CompanyDetail
- Provides the detailed view of a Company.
- Overrides the get_object method to return the Company that the user belongs to.
- We don’t need to create a CompanyList since there is no use-case to see a list of all the companies.

Note : We’re only checking for IsAuthenticated permission (except for account creation). This allows a User in the same company to modify/delete any user in the company (including themselves). This can be fixed by creating custom permissions class.

Let’s create our URLs. Create a file named urls.py in the accounts folder and add the following code:

from django.urls import path
from . import views

urlpatterns = [
    path('', views.AccountCreate.as_view(), name=views.AccountCreate.name),
    path('users/', views.UserList.as_view(), name=views.UserList.name),
    path('users/<uuid:pk>', views.UserDetail.as_view(), name=views.UserDetail.name),
    path('company', views.CompanyDetail.as_view(), name=views.CompanyDetail.name),
]

Open the main urls.py located in the django_saas folder and add the following code:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/v1/accounts/', include('accounts.urls')),

    path('api/v1/auth/', include('rest_framework.urls'))
]

We define our URLs and include the URLs for authentication from rest_framework. The URLs for our API look like this:

/api/v1/accounts/: We can only POST to this URL to create new account (company + user).
/api/v1/users/: Can call POST to create a new user for the company or GET to get all the users in the company.
/api/v1/users/<uuid>: Can call GET, PUT, PATCH and DELETE. GET returns a detailed view of the user with primary key <uuid>.
/api/v1/company: Can call GET and PUT, PATCH to get/update the company of the user that made the request.
/api/v1/auth/login: URL to log in using rest_framework’s built-in log in.

There is one more change I want to do before we test our accounts app. Open the settings.py file and add the following line:

# ...
LOGIN_REDIRECT_URL = '/api/v1/accounts/users'

Setting the LOGIN_REDIRECT_URL will redirect to this URL once a user has logged in successfully.

Let’s fire up the server. Go to your terminal and type in the following command:

$ python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).
October 17, 2018 - 06:05:27
Django version 2.1.2, using settings 'saas_django.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Our server is up! Now go to your favorite browser and navigate to http://127.0.0.1:8000/api/v1/accounts/. You should see something like this:

Note that the GET actually failed with a 405 Method Not Allowed. This is expected since this view should only be used for account creation. Note that the form clearly splits the Company and the User since we’re using the AccountSerializer.

Let’s create two accounts.

Account 1 information

Company Name : Company 1
Username : user1

Account 2 information

Company Name : Company 2
Username : user2

Input this information in the fields (select a password) and click the POST button. If all goes well, you should see something like this:

Note that the password doesn’t show up in the returned JSON since we set it to be write-only in the UserSerializer.

Let’s log in as user1. Navigate to http://127.0.0.1:8000/api/v1/auth/login and use the credentials to log in as user1. You should see something like this:

We just see one user (user1) even though we created two. This shows our get_queryset defined in our views in action. Let’s add another user and make sure we can see that one. We’ll call this one user3. After creating the user, click the GET button and you should see both user1 and user3.

HTTP 200 OK
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

[
    {
        "url": "http://127.0.0.1:8000/api/v1/accounts/users/962e4475-e213-4485-b8da-34cd0b34040a",
        "id": "962e4475-e213-4485-b8da-34cd0b34040a",
        "username": "user1"
    },
    {
        "url": "http://127.0.0.1:8000/api/v1/accounts/users/896b5d68-1215-43cd-9ad5-182518cf1dcd",
        "id": "896b5d68-1215-43cd-9ad5-182518cf1dcd",
        "username": "user3"
    }
]

Take note of the URLs for the next experiment. Let’s log in as user2 and let’s try to visit one of the URLs of a user belonging to Company 1. You should see something like this:

This again shows our get_queryset method in action. Even though a user with that uuid exists, it isn’t related to the company that user2 belongs to, so our queryset filters it out.

Let’s run one more experiment. Logged in as user2 point your browser to http://127.0.0.1:8000/api/v1/accounts/company. You should see something like this:

Company 2 is returned since that is the company that user2 belongs to. This shows our get_object method in action in the CompanyDetail view.

While we’re logged in as user2, let’s create user4 and user6 (let’s stick to even numbers for this company). Then, log in as either user1 or user3 and create user5.

The User Messages App

Let’s create another app called user_messages (I tried to call it messages earlier, but it conflicted with one of Django’s internal apps).

$ python manage.py startapp user_messages
$

Before we start with our UserMessage model, let’s create a model that handles the relationship with the Company model. For this, let’s create a new directory named core and let’s add a models.py there. The structure should look something like this:

$ tree | grep -v pyc
.
├── accounts
│   ├── __init__.py
│   ├── ...
├── core
│   ├── __init__.py
│   └── models.py
├── manage.py
├── saas_django
│   ├── __init__.py
│   ├── ...
└── user_messages
    ├── __init__.py
    ├── ...

9 directories, 38 files

Let’s add the following code to core/models.py:

import uuid
from django.db import models

class CompanyRelatedModel(models.Model):
    """Abstract class used by models that belong to a Company"""

    id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    company = models.ForeignKey('accounts.Company', related_name='%(class)s', on_delete=models.CASCADE, editable=False)

    class Meta:
        abstract = True

Here we define an abstract class that has a foreign key to accounts.Company. Let’s use this class by adding the following code to user_messages/models.py:

from django.contrib.auth import get_user_model
from django.db import models

from core.models import CompanyRelatedModel

User = get_user_model()

class UserMessage(CompanyRelatedModel):
    text = models.TextField('message', blank=False, null=False)
    date = models.DateTimeField('date', auto_now_add=True)
    from_user = models.ForeignKey(User, related_name='sent_messages', on_delete=models.CASCADE)
    to_user = models.ForeignKey(User, related_name='received_messages', on_delete=models.CASCADE)

    class Meta:
        db_table = 'user_messages'
        ordering = ['date']

We define a UserMessage class that inherits from CompanyRelatedModel. The model includes the following fields:

id: This field comes from the CompanyRelatedModel.
company: This field comes from the CompanyRelatedModel. It relates a user_message to a company.
text: The text of the user message.
date: The date and time when the message was sent.
from_user: A foreign key to the user that sent the message.
to_user: A foreign key to the user that is the recipient of the message.

Let’s open our settings.py and add our app to the INSTALLED_APPS list:

INSTALLED_APPS = [
    ...

    'accounts',

    # Add this line
    'user_messages',
]

Let’s run the migrations:

$ python manage.py makemigrations
Migrations for 'user_messages':
  user_messages/migrations/0001_initial.py
    - Create model UserMessage
$ python manage.py migrate
Operations to perform:
  Apply all migrations: accounts, admin, auth, contenttypes, sessions, user_messages
Running migrations:
  Applying user_messages.0001_initial... OK

Let’s create a serializers.py file in our user_messages app and add the following code:

from rest_framework import serializers
from .models import UserMessage

class UserMessageSerializer(serializers.HyperlinkedModelSerializer):
    class Meta:
        model = UserMessage
        fields = (
            'id',
            'url',
            'from_user',
            'to_user',
            'text',
            'date',
        )

We create a UserMessageSerializer with the desired fields. We will make some modifications later.

Let’s open our views.py and add the following code:

from rest_framework import permissions
from rest_framework import generics

from . import serializers
from .models import UserMessage

class UserMessageList(generics.ListCreateAPIView):
    name = 'usermessage-list'
    permission_classes = (
        permissions.IsAuthenticated,
    )
    serializer_class = serializers.UserMessageSerializer
    queryset = UserMessage.objects.all()

    def perform_create(self, serializer):
        company_id = self.request.user.company_id
        serializer.save(company_id=company_id)

    def get_queryset(self):
        company_id = self.request.user.company_id
        return super().get_queryset().filter(company_id=company_id)

class UserMessageDetail(generics.RetrieveAPIView):
    name = 'usermessage-detail'
    permission_classes = (
       permissions.IsAuthenticated,
    )
    serializer_class = serializers.UserMessageSerializer
    queryset = UserMessage.objects.all()

    def get_queryset(self):
        company_id = self.request.user.company_id
        return super().get_queryset().filter(company_id=company_id)

We define a UserMessageList view and a UserMessageDetail view. A few things to note:

Both views are guarded by the IsAuthenticated permission.
The perform_create function ensures that the UserMessage is related to the company that the current logged in user belongs to.
The UserMessageDetail extends the generics.RetrieveAPIView class. This means that a UserMessage is not editable or deletable (can only be viewed).
The get_queryset method is the same in both of these classes as well as in the UserList and UserDetail in the accounts application. We’ll fix this later.
The get_queryset method makes sure that users can only see messages within their company. However, a user can still see messages from other users in their company. We’ll fix this later.

Let’s create the urls.py for the user_messages app and add the following code:

from django.urls import path
from . import views

urlpatterns = [
    path('', views.UserMessageList.as_view(), name=views.UserMessageList.name),
    path('<uuid:pk>', views.UserMessageDetail.as_view(), name=views.UserMessageDetail.name),
]

Let’s include those urls in the main urls.py:

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/v1/accounts/', include('accounts.urls')),
    # Add this line
    path('api/v1/user-messages/', include('user_messages.urls')),

    path('api/v1/auth/', include('rest_framework.urls'))
]

After starting the server, navigate to http://127.0.0.1:8000/api/v1/user-messages/. You should see something like this:

Great! We got it to work!

Refactoring the serializer

Now, on to the bad news, expand the From User select field. You should see something like this:

Note that we see users from both companies. To fix it, let’s create a serializers.py in core and add the following code:

from rest_framework import serializers

class CompanySafeRelatedField(serializers.HyperlinkedRelatedField):
    """
    Ensures that the queryset only returns values for the company
    """
    def get_queryset(self):
        request = self.context['request']
        company_id = request.user.company_id
        return super().get_queryset().filter(company_id=company_id)

class CompanySafeSerializerMixin(object):
    """
    Mixin to be used with HyperlinkedModelSerializer to ensure that only company values are returned
    """
    serializer_related_field = CompanySafeRelatedField

We create two classes:

CompanySafeRelatedField:
- Extends the default HyperlinkedRelatedField
- Overrides the get_queryset method to filter the results by the company of the logged in user.
CompanySafeSerializerMixin:
- Assigns the previously defined CompanySafeRelatedField to the serializer_related_field.
- You can read more about serializer_related_field here.

Open user_messages/serializers.py and change it to be the following:

from rest_framework import serializers
# Changed line (adds the import of CompanySafeSerializerMixin)
from core.serializers import CompanySafeSerializerMixin
from .models import UserMessage

# Changed line (adds the CompanySafeSerializerMixin)
class UserMessageSerializer(CompanySafeSerializerMixin, serializers.HyperlinkedModelSerializer):
    class Meta:
        model = UserMessage
        fields = (
            'id',
            'url',
            'from_user',
            'to_user',
            'text',
            'date',
        )

Let’s head back to our browser and refresh. Your user list should only display users from the same company:

We have isolated users from other companies!

Note : A strange thing about this API is that the user needs to specify the From user. This can lead to users sending messages on behalf of other users. Since the user that is sending the message is already logged in, there is no point for the From user to be specified (the API should take care of this). We will fix this later.

Refactoring the views

Let’s abstract the Company-specific part of the views. Create core/views.py and add the following code:

from django.core import exceptions

class CompanySafeViewMixin:
    """
    Mixin to be used with views that ensures that models are related to the company during creation and are querysets
    are filtered for read operations
    """
    def get_queryset(self):
        queryset = super().get_queryset()

        if not self.request.user.is_authenticated:
            raise exceptions.NotAuthenticated()

        company_id = self.request.user.company_id
        return queryset.filter(company_id=company_id)

    def perform_create(self, serializer):
        company_id = self.request.user.company_id
        serializer.save(company_id=company_id)

This code abstracts the get_queryset and the perform_create methods since they’re pretty similar for most views.

Let’s add our mixin to our views. Open accounts/views.py and add the following code:

from rest_framework import permissions
from rest_framework import generics
from django.contrib.auth import get_user_model

# New import
from core.views import CompanySafeViewMixin
from . import serializers

User = get_user_model()

class AccountCreate(generics.CreateAPIView):
    name = 'account-create'
    serializer_class = serializers.AccountSerializer

# Added CompanySafeViewMixin
class UserList(CompanySafeViewMixin, generics.ListCreateAPIView):
    name = 'user-list'
    permission_classes = (
        permissions.IsAuthenticated,
    )
    serializer_class = serializers.UserSerializer
    queryset = User.objects.all()

    # Removed get_queryset and perform_create

# Added CompanySafeViewMixin
class UserDetail(CompanySafeViewMixin, generics.RetrieveUpdateDestroyAPIView):
    name = 'user-detail'
    permission_classes = (
       permissions.IsAuthenticated,
    )
    serializer_class = serializers.UserSerializer
    queryset = User.objects.all()

    # Removed get_queryset

class CompanyDetail(generics.RetrieveUpdateAPIView):
    name = 'company-detail'
    permission_classes = (
        permissions.IsAuthenticated,
    )
    serializer_class = serializers.CompanySerializer

    def get_object(self):
        return self.request.user.company

We inherit from the CompanySafeViewMixin that we defined before to our UserList and UserDetail views. We also remove the get_queryset and perform_create since these are already implemented in the CompanySafeViewMixin.

Preventing user impersonation & protecting user privacy

We could use the CompanySafeViewMixin in our user_messages app as well, and the current functionality would still work the same. However, we have two problems with the API right now:

A user could impersonate another user because the From user can be specified when we create a message.
Users could read all of the messages related to their Company even if they weren’t the senders or recipients.

To fix these two problems, we need to tune our own perform_create and get_queryset to meet our needs. Let’s make a few changes in our UserMessage model first. Open user_messages/models.py and add the following code:

from django.contrib.auth import get_user_model
from django.db import models
# New import
from django.db.models import Q

from core.models import CompanyRelatedModel

User = get_user_model()

# New class
class UserMessageManager(models.Manager):

    def get_for_user(self, user):
        """Retrieves all messages that a user either sent or received"""
        return self.all().filter(company_id=user.company_id).filter(Q(from_user=user) | Q(to_user=user))

class UserMessage(CompanyRelatedModel):
    text = models.TextField('message', blank=False, null=False)
    date = models.DateTimeField('date', auto_now_add=True)
    from_user = models.ForeignKey(User, related_name='sent_messages', on_delete=models.CASCADE)
    to_user = models.ForeignKey(User, related_name='received_messages', on_delete=models.CASCADE)

    # New model manager
    objects = UserMessageManager()

    class Meta:
        db_table = 'user_messages'
        ordering = ['date']

We create a new model manager called UserMessageManager and create a method get_for_user. This method first filters by the company_id to make sure nothing ever gets returned from other companies. Then, it filters to get messages that were sent from or to the user passed in as the argument.

Let’s open user_messages/serializers.py and add the following code:

from rest_framework import serializers
from core.serializers import CompanySafeSerializerMixin
from .models import UserMessage

class UserMessageSerializer(CompanySafeSerializerMixin, serializers.HyperlinkedModelSerializer):
    class Meta:
        model = UserMessage
        fields = (
            'id',
            'url',
            'from_user',
            'to_user',
            'text',
            'date',
        )
        # New read_only field
        read_only_fields = (
            'from_user',
        )

We make the from_user a read-only field. That way, our API will still return the sender of a message but won’t allow it to be edited. Since the from_user field can’t be specified by the client, we need to make a change in the view. Open user_messages/views.py and add the following code:

from rest_framework import permissions
from rest_framework import generics

from . import serializers
from .models import UserMessage

class UserMessageList(generics.ListCreateAPIView):
    name = 'usermessage-list'
    permission_classes = (
        permissions.IsAuthenticated,
    )
    serializer_class = serializers.UserMessageSerializer
    queryset = UserMessage.objects.all()

    def perform_create(self, serializer):
        user = self.request.user
        company_id = user.company_id
        # Added from_user
        serializer.save(company_id=company_id, from_user=user)

    def get_queryset(self):
        # Changed this to use the UserMessageManager's method
        return UserMessage.objects.get_for_user(self.request.user)

class UserMessageDetail(generics.RetrieveAPIView):
    name = 'usermessage-detail'
    permission_classes = (
       permissions.IsAuthenticated,
    )
    serializer_class = serializers.UserMessageSerializer

    def get_queryset(self):
        # Changed this to use the UserMessageManager's method
        return UserMessage.objects.get_for_user(self.request.user)

This fixes the two issues mentioned above. We change the perform_create function to include the from_user in the serializer.save call. We pass in the user that is currently logged in. We also change the get_queryset method to only return messages that were either sent or received by the current user.

Testing it out

Let’s log in as user3 and navigate to http://127.0.0.1:8000/api/v1/user-messages/. Let’s send a message to user1:

Note that there is no longer a choice to set the From user. Let’s press the POST button and verify that the message gets created with from_user as user3:

Note that the from_user field is a URL. Let’s click on that URL to make sure it’s user3.

Let’s log in as user1 in a different browser and reply to user3. The messages should be visible for both user1 and user3.

As expected, the messages can be seen by both user1 and user3. Let’s log in as user5 and navigate to http://127.0.0.1/api/v1/user-messages/. We shouldn’t be able to see any of the messages that we created.

As expected, user5 sees an empty list since no messages have been sent to/by this user.

Further improvements

We’re now done with a simple use-case. Here are some ways to improve this further:

Add real versioning support for the API.
Improve the directory structure to have a more cookiecutter approach.
Remove SQLite and use Postgres.
Prevent users from deleting themselves (can be done using permissions).
Validate that users don’t message themselves.
Prevent users from modifying other users (can be done using permissions).
Add more models and views to use the CompanyRelatedModel and CompanySafeViewMixin.
Add filtering for user messages to be able to filter by from_user, to_user, date, etc (can be done using django-filter).
Add the possibility to mark a message as read.

Conclusion

We went through the process of creating a SaaS messaging application that uses the same database schema for everybody. We went through incremental features and improvements to get to a final product that implemented all of our use-cases. We also abstracted out the repetitive code to make it more maintainable.

How to Access Logged Objects in the Javascript Console

Nicolas Mesa — Mon, 24 Sep 2018 04:54:51 +0000

Cross-posted from my personal blog https://blog.nicolasmesa.co.

In this short post, we go through the process of making a console.logged Object available in Chrome’s javascript console.

TLDR

Right-click the Object you want to inspect and click Store as global variable. This creates a variable with name temp<n> where <n> is a number (starts from 1). Use the variable to access the Object.

Explanation

The Hack Version

Sometimes when I’m debugging, I need to access an Object and its methods from the Javascript console. To do this, I have to fire up the debugger and step into the function until I get access to the object I want, or I have to come up with a hack like the following:

function superNestedFunction() {
    const obj = getObj();
    console.log(obj);

    // hack
    window.myObj = obj;

    // Do something interesting with obj
}

For this discussion, let’s say the getMyObj() function returns an object like this:

function getObj() {
    return {
        prop: 'Hello World',
        fn: function() {
            console.log('Inside myObj.fn');
        }
    };
}

Then, from the console, I can do:

superNestedFunction(); // Logs: {prop: "Hello World", fn: ƒ}
window.myObj.prop; // Logs: "Hello World"
window.myObj.fn(); // Logs: Inside myObj.fn

Note that this is a terrible hack! As much as I hate having a bunch of objects in my logs, I think it’s a lot worse to pollute the global namespace (especially when you forget to remove that hack). Let’s look at a better way of doing this.

The better version

We’re going to use the same example that we had before (minus the hack). The entire code looks like this:

function superNestedFunction() {
    const obj = getObj();
    console.log(obj); // We're still logging the object

    // Do something interesting with obj
}

function getObj() {
    return {
        prop: 'Hello World',
        fn: function() {
            console.log('Inside myObj.fn');
        }
    };
}

superNestedFunction(); // Logs: {prop: "Hello World", fn: ƒ}

Now right-click on the logged Object and select the Store as global variable option.

Chrome automatically creates a new global temporary variable (temp1 in this case) and assigns it the Object that you right-clicked:

Now you can use temp1 to access any field of that Object! Let’s see the whole thing in action:

Conclusion

We went through the steps to assign a logged object to a global variable and were able to access its properties and functions. Note that this is not a replacement for the debugger, it’s just a convenient way to explore an Object. Especially if the console.log is already there.

Happy debugging!

How Does Traceroute Work

Nicolas Mesa — Mon, 17 Sep 2018 03:50:26 +0000

Cross-posted from my personal blog https://blog.nicolasmesa.co.

In this post, we take a look at how the traceroute command works. traceroute is a utility command that prints the route (or hops) that a packet takes to reach another host. We start with an example of traceroute. Then we go through what happened behind the scenes. Finally, we run the traceroute command one more time while sniffing the traffic with tcpdump.

Traceroute example

Let’s look at an example:

nmesa@desktop-nicolas:~$ ping -c 1 google.com
PING google.com (172.217.14.206) 56(84) bytes of data.
64 bytes from sea30s01-in-f14.1e100.net. (172.217.14.206): icmp_seq=1 ttl=56 time=8.02 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 8.028/8.028/8.028/0.000 ms

nmesa@desktop-nicolas:~$ traceroute -q 1 google.com
traceroute to google.com (172.217.14.206), 64 hops max
  1 192.168.0.1 0.247ms
  2 67.218.102.107 11.667ms
  3 174.127.182.56 10.964ms
  4 174.127.141.190 12.403ms
  5 72.14.198.216 11.208ms
  6 74.125.243.177 13.321ms
  7 209.85.254.237 11.483ms
  8 172.217.14.206 12.060m

First, we issue a ping to google.com to find out its IP address (172.217.14.206). Then, we run a traceroute command to google.com (the -q 1 is to have traceroute only send one packet instead of 3). In this example, we see that the packets have to traverse 8 hops to reach google.com’s host.

IP Header

So, how does this work under the hood? To understand this, first, we need to talk about the Internet Protocol (IP). IP is responsible for delivering packets from the source host to the destination. Every IP packet has an IP header, which is used by routers to make routing decisions. The IP header has a lot of information, but we’re going to focus on 3 fields:

Source IP (src): The IP address of the host that sent the packet (we’re going to ignore NATs for this discussion). In our case, it’s 192.168.0.250.
Destination IP (dst): IP address of the target host. In our case, it’s google.com’s IP address: 172.217.14.206.
Time to Live field (TTL): This field doesn’t contain a time, as the name suggests. It has the maximum number of hops that this packet should traverse before reaching its destination. If the maximum number of hops is reached, the packet is dropped, and an error message is sent to the sender. This field helps to prevent routing loops where a packet stays in a loop forever.

When a router receives an IP packet, it subtracts 1 from TTL field to determine if it should forward the packet or not. If the new TTL value is greater than 0, the router forwards the packet with the updated TTL value. If the new TTL is equal to 0, the router discards the packet and sends an ICMP error message to the source to let it know that the packet’s time exceeded in-transit (the TTL wasn’t large enough for the packet to reach the destination).

Traceroute introduction

traceroute takes advantage of the TTL field and the ICMP error message returned to trace the route to the target host. First, it sends a packet (usually UDP if in Linux) with TTL = 1. When this packet reaches our router, the router decrements the TTL and notices that the TTL is equal to 0. Our router drops the packet and sends back an ICMP error message. traceroute uses the source IP address in the ICMP packet as the value of the first hop. traceroute sends another datagram with TTL = 2. Our router gets the packet, decrements the TTL and forwards the packet since the TTL is greater than 0. The second hop receives the packet, decrements the TTL and notices that it is equal to 0. The router drops the packet and sends back an ICMP error message which traceroute uses to get the value for the second hop. traceroute keeps incrementing the TTL by one until it actually reaches the server. The server responds typically with an ICMP error stating that the UDP port is unreachable. At that point, traceroute is done.

Capturing the traceroute traffic

Let’s use tcpdump to capture this exchange. First, we open a new terminal window and execute the following tcpdump command:

nmesa@desktop-nicolas:~$ sudo tcpdump -vv -nn -i eth0 "(udp and ip[8] < 10) or icmp"
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

Let’s go over the arguments:

-vv: To make the output verbose. We need this to be able to see the packet’s TTL value.
-nn: We don’t want tcpdump to resolve the IP addresses to hostnames or ports to service names.
-i eth0: The interface to listen on.
"(udp and ip[8] < 10) or icmp": Filter to display either UDP packets with a TTL value of less than 10 (adjust this number to meet your needs) or ICMP packets (to see the errors coming back from the routers).
- The ip[8] notation means the 8th byte (starting from 0) of the IP Header.

In another terminal window, we run the traceroute command:

nmesa@desktop-nicolas:~$ traceroute -q 1 google.com
traceroute to google.com (172.217.14.206), 64 hops max
  1 192.168.0.1 0.247ms
  2 67.218.102.107 11.667ms
  3 174.127.182.56 10.964ms
  4 174.127.141.190 12.403ms
  5 72.14.198.216 11.208ms
  6 74.125.243.177 13.321ms
  7 209.85.254.237 11.483ms
  8 172.217.14.206 12.060m

If we go back to the tcpdump window, we see something like this:

nmesa@desktop-nicolas:~$ sudo tcpdump -vv -nn -i eth0 "(udp and ip[8] < 10) or icmp"
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:48:11.140870 IP (tos 0x0, ttl 1, id 34332, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33434: [bad udp cksum 0x7d6c -> 0x0386!] UDP, length 9
21:48:11.141090 IP (tos 0xc0, ttl 64, id 59363, offset 0, flags [none], proto ICMP (1), length 65)
    192.168.0.1 > 192.168.0.250: ICMP time exceeded in-transit, length 45
        IP (tos 0x0, ttl 1, id 34332, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33434: [udp sum ok] UDP, length 9
21:48:11.141236 IP (tos 0x0, ttl 2, id 34333, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33435: [bad udp cksum 0x7d6c -> 0x0385!] UDP, length 9
21:48:11.152912 IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    67.218.102.107 > 192.168.0.250: ICMP time exceeded in-transit, length 36
        IP (tos 0x0, ttl 1, id 34333, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33435: UDP, length 9
21:48:11.152943 IP (tos 0x0, ttl 3, id 34335, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33436: [bad udp cksum 0x7d6c -> 0x0384!] UDP, length 9
21:48:11.163898 IP (tos 0x0, ttl 253, id 26888, offset 0, flags [none], proto ICMP (1), length 96)
    174.127.182.56 > 192.168.0.250: ICMP time exceeded in-transit, length 76
        IP (tos 0x0, ttl 1, id 34335, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33436: [udp sum ok] UDP, length 9
21:48:11.163934 IP (tos 0x0, ttl 4, id 34336, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33437: [bad udp cksum 0x7d6c -> 0x0383!] UDP, length 9
21:48:11.176324 IP (tos 0x0, ttl 252, id 59054, offset 0, flags [none], proto ICMP (1), length 96)
    174.127.141.190 > 192.168.0.250: ICMP time exceeded in-transit, length 76
        IP (tos 0x0, ttl 1, id 34336, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33437: [udp sum ok] UDP, length 9
21:48:11.176368 IP (tos 0x0, ttl 5, id 34339, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33438: [bad udp cksum 0x7d6c -> 0x0382!] UDP, length 9
21:48:11.187557 IP (tos 0x0, ttl 251, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    72.14.198.216 > 192.168.0.250: ICMP time exceeded in-transit, length 36
        IP (tos 0x0, ttl 1, id 34339, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33438: UDP, length 9
21:48:11.187622 IP (tos 0x0, ttl 6, id 34340, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33439: [bad udp cksum 0x7d6c -> 0x0381!] UDP, length 9
21:48:11.200912 IP (tos 0x0, ttl 249, id 45260, offset 0, flags [none], proto ICMP (1), length 96)
    74.125.243.177 > 192.168.0.250: ICMP time exceeded in-transit, length 76
        IP (tos 0x80, ttl 1, id 34340, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33439: [udp sum ok] UDP, length 9
21:48:11.201012 IP (tos 0x0, ttl 7, id 34341, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33440: [bad udp cksum 0x7d6c -> 0x0380!] UDP, length 9
21:48:11.212452 IP (tos 0x0, ttl 248, id 36434, offset 0, flags [none], proto ICMP (1), length 96)
    209.85.254.237 > 192.168.0.250: ICMP time exceeded in-transit, length 76
        IP (tos 0x80, ttl 1, id 34341, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33440: [udp sum ok] UDP, length 9
21:48:11.212596 IP (tos 0x0, ttl 8, id 34342, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33441: [bad udp cksum 0x7d6c -> 0x037f!] UDP, length 9
21:48:11.224616 IP (tos 0x0, ttl 56, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    172.217.14.206 > 192.168.0.250: ICMP 172.217.14.206 udp port 33441 unreachable, length 36
        IP (tos 0x80, ttl 1, id 34342, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33441: UDP, length 9

Traceroute traffic deep dive

Let’s break this down. Line number 3 in the traceroute command shows that the first hop is 192.168.0.1 (my router). Take a look at the following line:

21:48:11.140870 IP (tos 0x0, ttl 1, id 34332, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33434: [bad udp cksum 0x7d6c -> 0x0386!] UDP, length 9
21:48:11.141090 IP (tos 0xc0, ttl 64, id 59363, offset 0, flags [none], proto ICMP (1), length 65)
    192.168.0.1 > 192.168.0.250: ICMP time exceeded in-transit, length 45
        IP (tos 0x0, ttl 1, id 34332, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33434: [udp sum ok] UDP, length 9

Note the ttl 1 value. We can also see the time exceeded in-transit message coming back with source IP 192.168.0.1. That corresponds with the first hop!

The second hop in the traceroute command is 67.218.102.107 (line 4). This also corresponds to the following lines:

21:48:11.141236 IP (tos 0x0, ttl 2, id 34333, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33435: [bad udp cksum 0x7d6c -> 0x0385!] UDP, length 9
21:48:11.152912 IP (tos 0x0, ttl 254, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    67.218.102.107 > 192.168.0.250: ICMP time exceeded in-transit, length 36
        IP (tos 0x0, ttl 1, id 34333, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33435: UDP, length 9

Here we see the ttl 2 value. Note that the source IP (192.168.0.250) and destination IP (172.217.14.206) haven’t changed (the target port increases by 1 every message, but this doesn’t affect our discussion). The ICMP error message has source IP 67.218.102.107 which also aligns with what we see in our traceroute command! Let’s go all the way to the end!

The last hop in the traceroute command is 172.217.14.206, one of the IP addresses of google.com. Let’s examine our tcpdump output for that hop.

21:48:11.212596 IP (tos 0x0, ttl 8, id 34342, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33441: [bad udp cksum 0x7d6c -> 0x037f!] UDP, length 9
21:48:11.224616 IP (tos 0x0, ttl 56, id 0, offset 0, flags [none], proto ICMP (1), length 56)
    172.217.14.206 > 192.168.0.250: ICMP 172.217.14.206 udp port 33441 unreachable, length 36
        IP (tos 0x80, ttl 1, id 34342, offset 0, flags [DF], proto UDP (17), length 37)
    192.168.0.250.50475 > 172.217.14.206.33441: UDP, length 9

We now see a ttl 8 value. Note that there isn’t a time exceeded in-transit ICMP error message. Instead, the server responded with an ICMP message saying udp port 33441 unreachable. This error is coming from the host since it is the only one who knows if that port is open or not. This message tells traceroute that the last packet it sent reached the host and there are no more hops in the way.

Conclusion

We covered how traceroute works under the hood. We saw how this command uses the TTL field to control the number of hops that the packet travels before it is dropped and an error message is sent back to us. We also saw that the source IP address of the ICMP error message is also the IP address of the hop that dropped the packet. Some routers are configured to drop the packet silently and not send an ICMP error message (this shows up as a * entry in the traceroute output).

One thing we didn’t talk about is the IP header checksum. The checksum is used to verify the contents of the IP header (not the IP body). This checksum is validated on every hop and needs to be recalculated after decrementing the TTL value.

Serve Your Current Directory With Python and HTTP

Nicolas Mesa — Mon, 03 Sep 2018 18:08:26 +0000

Cross-posted from my personal blog https://blog.nicolasmesa.co.

This is going to be a short post showing how to run an HTTP server to serve your current working directory.

TLDR

For Python 3 run:

python3 -m http.server

For Python 2 run:

python -m SimpleHTTPServer

.bashrc alias:

alias serve="python3 -m http.server"

Explanation

Sometimes at work, I’ve had the need to spin up a quick HTTP server to serve my current working directory. I usually need this for two use cases:

Share a quick link to have people download something from my computer.
I have an index.html file in another computer along with a bunch of static resources and don’t feel like copying the whole folder to be able to see the page in my browser. This use case happens a lot when I’m running unit tests on another computer and need to look at coverage reports (which are rendered in HTML).

It turns out that serving your current directory with an HTTP server is easy. You only need python.

Setup

Before we get started, let’s create some sample files.

nmesa@desktop-nicolas:~/demos/serve-cwd$ echo file1 > file1.txt
nmesa@desktop-nicolas:~/demos/serve-cwd$ echo file2 > file2.txt
nmesa@desktop-nicolas:~/demos/serve-cwd$ mkdir dir1 dir2
nmesa@desktop-nicolas:~/demos/serve-cwd$ echo file3 > dir1/file3.txt
nmesa@desktop-nicolas:~/demos/serve-cwd$ echo file4 > dir2/file4.txt
nmesa@desktop-nicolas:~/demos/serve-cwd$ tree
.
├── dir1
│   └── file3.txt
├── dir2
│   └── file4.txt
├── file1.txt
└── file2.txt

2 directories, 4 files

We created two files and two directories in the root, and one file in each of the directories.

Find out your python version

The command to spin up the HTTP server varies depending on the version of python that you have. Run python --version to get your current version of python.

nmesa@desktop-nicolas:~/demos/serve-cwd$ python --version
Python 3.5.2

If you see Python 3.x.x, use the Python 3 command. If you see Python 2.x.x, use the Python 2 command.

In this case, I’m using Python 3.

Python 3 command

nmesa@desktop-nicolas:~/demos/serve-cwd$ python3 -m http.server
Serving HTTP on 0.0.0.0 port 8000 ...

Python 2 command

nmesa@desktop-nicolas:~/demos/serve-cwd$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Test it out

The previous commands spun up an HTTP server listening on port 8000. Let’s point our web browser to the server’s IP address on port 8000 (192.168.0.250:8000 in this case). The page should look like this:

Root directory

Sample root directory listing. This is the equivalent of running an ls on the directory where the server is running.

Child directory

Let’s click the dir1/ link:

dir1 directory listing.

File

Let’s click the file3.txt link.

file3.txt contents. Note that this file rendered in the browser. This is because the HTTP server is smart enough to add a content type to the HTTP response. This becomes handy when you need to serve HTML along with CSS, JS, and images.

Update (Oct 21, 2018)

I’ve been using this a lot at work to serve test coverage reports. To save some typing, let’s add the following alias to our .bashrc (.bash_profile in Mac) file:

alias serve="python3 -m http.server"

Let’s test it out by running serve from our target directory (note that you will need to run source ~/.bashrc or start a new terminal for this to work):

nmesa@desktop-nicolas:~/demos/serve-cwd$ serve
Serving HTTP on 0.0.0.0 port 8000 ...

We can also specify a different port:

nmesa@desktop-nicolas:~/demos/serve-cwd$ serve 9000
Serving HTTP on 0.0.0.0 port 9000 ...

This command starts the web server in port 9000.

Conclusion

Python makes it easy to spin up an HTTP server to share your current working directory. Please don’t forget to kill the server once you’re done and be careful with what you share.

Links / Further reading

Container Creation Using Namespaces and Bash

Nicolas Mesa — Fri, 24 Aug 2018 04:30:27 +0000

Cross-posted from my personal blog https://blog.nicolasmesa.co.

A few weeks ago, I saw a great video explaining how Docker works under the hood (see video below). The video ends with a demo where Jérôme Petazzoni creates a container using nothing but bash. I found many of the commands that he used pretty cryptic, so I decided to explain what he did and the purpose of each command.

Video

(The demo starts around minute 41)

Terminology

Before we dive into the demo, let’s get some terminology out of the way.

Container

A container is a combination of a few technologies including namespaces, cgroups, and capabilities. In this post, we’re going to focus on namespaces.

Namespaces

From Namespaces in operation:

The purpose of each namespace is to wrap a particular global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of the global resource.

There are six types of namespaces in Linux:

Pid: Isolates process identifiers. For example, two processes in different namespaces can have the same PID.
User: Isolates user ids and group ids. Two users in two different user namespaces can have the same user ids. This is pretty useful because it allows mapping an unprivileged user id outside of the namespace to be root inside of the namespace.
Net: This namespace provides network isolation. Processes running in a separate net namespace don’t see the network interfaces of other namespaces.
Mnt: The mount namespace allows containers to have their own mount points without polluting the global namespace. It also provides a way to hide the global mount points from the container.
Uts: This allows a container to have its own hostname for the processes running in the container.
Ipc: Gives containers their own inter-process communication namespace.

Note : There is a lot more to namespaces than this. If you want to learn more, take a look at Namespaces in operation.

Btrfs

From the Btrfs wiki

Btrfs is a modern copy on write (CoW) filesystem for Linux aimed at implementing advanced features while also focusing on fault tolerance, repair, and easy administration.

Setup

System

For this demo, I used a computer running Ubuntu 16.04. To be more specific:

nmesa@desktop-nicolas:~/demos/containers$ uname -a
Linux desktop-nicolas 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:43:10 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

We also need docker for two things:

To extract the alpine docker image.
To use the bridge provided by docker for networking.

This is the version of docker that I had installed:

root@desktop-nicolas:/btrfs# docker version
Client:
 Version: 18.06.0-ce
 API version: 1.38
 Go version: go1.10.3
 Git commit: 0ffa825
 Built: Wed Jul 18 19:11:02 2018
 OS/Arch: linux/amd64
 Experimental: false

Server:
 Engine:
  Version: 18.06.0-ce
  API version: 1.38 (minimum version 1.12)
  Go version: go1.10.3
  Git commit: 0ffa825
  Built: Wed Jul 18 19:09:05 2018
  OS/Arch: linux/amd64
  Experimental: false

System setup

Before we start to follow along with the video, we to need setup our environment into the same state as Jérôme’s computer. We need to have a btrfs filesystem mounted in /btrfs.

Create the disk image

We start by creating an empty disk image (from the btrfs wiki, the image should be at least 1GB in size).

nmesa@desktop-nicolas:~/demos/containers$ dd if=/dev/zero of=disk.img bs=512 count=4194304
4194304+0 records in
4194304+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 4.60753 s, 466 MB/s

nmesa@desktop-nicolas:~/demos/containers$ ls -alh disk.img
-rw-rw-r-- 1 nmesa nmesa 2.0G Aug 23 22:06 disk.img

We use the dd command to create a 2GB empty image. The dd command receives four arguments:

if: The file/device to use as the input. In this case, /dev/zero, which outputs a bunch of zeroes.
of: The output file/device. In this case, disk.img.
bs: This is the block size in bytes (512 bytes in this case).
count: The number of bs blocks to read from /dev/zero and write to disk.img. In this case, 4194304 (2 * 1024 * 1024 * 1024 / 512) to make the image 2GB.

Format disk image

Let’s format our new image with the btrfs filesystem by using the mkfs.btrfs command.

nmesa@desktop-nicolas:~/demos/containers$ mkfs.btrfs disk.img
btrfs-progs v4.4
See http://btrfs.wiki.kernel.org for more information.

Label: (null)
UUID: 772f0676-ec96-448d-b58c-31d4c10b5c3a
Node size: 16384
Sector size: 4096
Filesystem size: 2.00GiB
Block group profiles:
  Data: single 8.00MiB
  Metadata: DUP 110.38MiB
  System: DUP 12.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Number of devices: 1
Devices:
   ID SIZE PATH
    1 2.00GiB disk.img

Mount the disk image

Great! We have formatted the filesystem! We need to mount the image to /btrfs using the mount command (note that you have to be root ).

root@desktop-nicolas:/home/nmesa/demos/containers# mkdir /btrfs
root@desktop-nicolas:/home/nmesa/demos/containers# mount -t btrfs disk.img /btrfs

We are ready to follow along with the video!

Video breakdown

In this section, we follow along with the video, and I describe what each command does. Most commands require that you run them as root.

Make mount private

root@desktop-nicolas:/# mount --make-rprivate /

With this command, we make the mounts that our container is going to make private. This prevents any mounts that our container does, from being visible by the host system. The r in --make-rprivate stands for recursive which means it makes all mount points private.

Create container image

Let’s create two directories, one to hold the source images and one to hold the container images.

root@desktop-nicolas:/btrfs# mkdir images containers

Then we use the btrfs subvol create command to create a new image called alpine.

root@desktop-nicolas:/btrfs# btrfs subvol create images/alpine
Create subvolume 'images/alpine'
root@desktop-nicolas:/btrfs# ls images/alpine/
root@desktop-nicolas:/btrfs#

The btrfs command is used to control btrfs filesystems. The subvolume subcommand is used to manage btrfs subvolumes. We use the create subcommand to create a subvolume named alpine in the images directory.

We have an image, but it’s empty. We use docker to get the alpine image.

root@desktop-nicolas:/btrfs# CID=$(docker run -d alpine true)
root@desktop-nicolas:/btrfs# echo $CID
9b290758c1734811c85445640c985fed9803121891d6604b4260e985c2647404
root@desktop-nicolas:/btrfs# docker export $CID | tar -C images/alpine/ -xf-
root@desktop-nicolas:/btrfs# ls images/alpine/
bin dev etc home lib media mnt proc root run sbin srv sys tmp usr var

We run docker run -d alpine true and assign its output (the container id) to the CID variable. Here’s a better explanation of the docker command:

run: runs a container.
-d: Detaches the container and prints the container id. Our terminal doesn’t wait for the container to finish its execution.
alpine: The image that the container should use.
true: The command to run in the container.

We use the docker export command to export the image. We pipe that image to the tar command which decompresses it and puts the files in images/alpine. Then we run ls images/alpine to see the contents of the image.

The next step is to take a snapshot of this image and put it in our containers folder. The snapshot’s name is tupperware.

root@desktop-nicolas:/btrfs# btrfs subvol snapshot images/alpine/ containers/tupperware
Create a snapshot of 'images/alpine/' in 'containers/tupperware'
root@desktop-nicolas:/btrfs# ls containers/tupperware/
bin dev etc home lib media mnt proc root run sbin srv sys tmp usr var

We used the snapshot subcommand to create a snapshot of the subvolume alpine with the name tupperware in the containers directory. Snapshots are very efficient. They don’t copy the data from the source subvolume making them space efficient and very fast to create. If a snapshot is modified, the original subvolume won’t be affected. We can see this in the following example:

root@desktop-nicolas:/btrfs# touch containers/tupperware/NICK_WAS_HERE
root@desktop-nicolas:/btrfs# ls containers/tupperware/
bin dev etc home lib media mnt NICK_WAS_HERE proc root run sbin srv sys tmp usr var
root@desktop-nicolas:/btrfs# ls images/alpine/
bin dev etc home lib media mnt proc root run sbin srv sys tmp usr var

In this example, we add a file called NICK_WAS_HERE to the snapshot. Then we prove that it is visible in the snapshot but not in the original alpine subvolume. More information about the btrfs command can be found in this post.

Testing using chroot

Let’s use the chroot command to run sh inside the tupperware snapshot.

root@desktop-nicolas:/btrfs# chroot containers/tupperware/ sh
/ # ls /
NICK_WAS_HERE dev home media proc run srv tmp var
bin etc lib mnt root sbin sys usr
/ # apk
apk-tools 2.10.0, compiled for x86_64.

Installing and removing packages:
  add Add PACKAGEs to 'world' and install (or upgrade) them, while ensuring that all dependencies are met
  del Remove PACKAGEs from 'world' and uninstall them

System maintenance:
  fix Repair package or upgrade it without modifying main dependencies
  update Update repository indexes from all remote repositories
  upgrade Upgrade currently installed packages to match repositories
  cache Download missing PACKAGEs to cache and/or delete unneeded files from cache

Querying information about packages:
  info Give detailed information about PACKAGEs or repositories
  list List packages by PATTERN and other criteria
  dot Generate graphviz graphs
  policy Show repository policy for packages

Repository maintenance:
  index Create repository index file from FILEs
  fetch Download PACKAGEs from global repositories to a local directory
  verify Verify package integrity and signature
  manifest Show checksums of package contents

Use apk <command> --help for command-specific help.
Use apk --help --verbose for a full command listing.

This apk has coffee making abilities.
/ # exit
root@desktop-nicolas:/btrfs#

chroot is used to change the root directory of a program. In the example above, the /btrfs/container/tupperware directory becomes the root of the filesystem. We execute apk, which is only available in alpine, and then we exit the container.

Note that we didn’t use namespaces in this “container.” Let’s change that!

Using namespaces

root@desktop-nicolas:/btrfs# unshare --mount --uts --ipc --net --pid --fork bash
root@desktop-nicolas:/btrfs#

We use the unshare command to run a program (bash) in different mount, uts, ipc, net and pid namespaces. We also pass in the --fork flag to create a new process as a child of the unshare command. It may seem like nothing happened from the command output, but we’re using namespaces and are somewhat isolated from the global namespace. Let’s change the hostname to prove that we are in an isolated uts namespace.

root@desktop-nicolas:/btrfs# hostname tupperware
root@desktop-nicolas:/btrfs# exec bash
root@tupperware:/btrfs#

Note that after executing bash, our hostname switched to tupperware. Running hostname from another terminal shows that the global namespace still has the same hostname (desktop-nicolas in this case).

# from a different terminal
nmesa@desktop-nicolas:~/demos/containers$ hostname
desktop-nicolas

Let’s do some experiments with the pid namespace.

root@tupperware:/btrfs# ps
  PID TTY TIME CMD
24506 pts/26 00:00:00 bash
24551 pts/26 00:00:00 unshare
24552 pts/26 00:00:00 bash
24961 pts/26 00:00:00 ps
28780 pts/26 00:00:00 sudo
28781 pts/26 00:00:00 su
28782 pts/26 00:00:00 bash
root@tupperware:/btrfs# pidof unshare
24551
root@tupperware:/btrfs# kill $(pidof unshare)
bash: kill: (24551) - No such process

We see a lot more processes than expected. The PIDs are also those of the main system. What is going on? It turns out that the /proc of the global namespace is mounted, and the ps command uses /proc to get some of its information. (You can verify this by running strace ps.)

Mount proc

Let’s mount /proc.

root@tupperware:/btrfs# mount -t proc proc /proc
root@tupperware:/btrfs# ps
  PID TTY TIME CMD
    1 pts/26 00:00:00 bash
   30 pts/26 00:00:00 ps
root@tupperware:/btrfs# umount /proc
root@tupperware:/btrfs#

After we mount proc, ps only shows the processes running in our container with the correct PIDs. We unmount /proc again for now.

We’ve used mount before. Let’s go over what it does.

The mount command is used to mount filesystems and usually has the following syntax:

mount -t <type> <device> <directory>

From the mount man page:

this tells the kernel to attach the filesystem found on <device> (which is of type <type>) at the directory <directory>.

Here’s the argument breakdown for our specific case:

-t proc: The type of filesystem to use is proc.
proc: The /proc filesystem is not attached to a device. Any keyword here would work.
- Note that in the video Jérôme uses none instead of proc for this value. I changed this to be proc because the mount man page discourages the use of none for this (“The customary choice none is less fortunate: the error message ‘none busy’ from umount can be confusing.“).
/proc: The directory where we want to mount the proc filesystem.

Pivot root

Let’s make /btrfs/containers/tupperware the root of our filesystem by using mount and pivot_root.

root@tupperware:/btrfs# mkdir /btrfs/containers/tupperware/oldroot
root@tupperware:/btrfs# mount --bind /btrfs/containers/tupperware /btrfs
root@tupperware:/btrfs# cd /btrfs/
root@tupperware:/btrfs# ls
bin dev etc home lib media mnt NICK_WAS_HERE oldroot proc root run sbin srv sys tmp usr var
root@tupperware:/btrfs# pivot_root . oldroot
root@tupperware:/btrfs# cd /
root@tupperware:/# ls
NICK_WAS_HERE dev home media oldroot root sbin sys usr
bin etc lib mnt proc run srv tmp var
root@tupperware:/# ls oldroot/
bin cdrom hello lib media root srv usr
boot core home lib32 mnt run sys var
bt dev initrd.img lib64 opt sbin tes vmlinuz
btrfs etc initrd.img.old lost+found proc snap tmp vmlinuz.old

Let’s break this down:

We create a directory called oldroot where our current root filesystem will be mounted. As Jérôme explained in the video, to be able to run pivot_root successfully, you need to be close to the top of the filesystem (/).
We use mount --bind to get the container filesystem to /btrfs. (Jérôme splits this into two steps, and I didn’t understand why. I decided to try it this way to see if it would work and it did). The mount --bind command is useful to mount a directory somewhere else (in our case we’re mounting /btrfs/containers/tupperware to /btrfs).
We go to /btrfs and run ls to make sure that our container filesystem is mounted there.
We execute pivot_root which switches the current directory (/btrfs) to be the new root and mounts the current root in oldroot.
We move to the new root and check that it is mounted correctly by running ls.
We verify that oldroot has the old root mount point.

Note : I didn’t understand why we needed to do a pivot_root instead of doing a bind mount point to /. This email by Linus Torvalds himself helped me understand. In a nutshell, / points to the process’ root of the filesystem. By doing a bind mount to root (/), the process would still point to the old version of /.

Fixing the mount points

Let’s take a look at our mount points.

root@tupperware:/# mount -t proc proc /proc
root@tupperware:/# mount | head
/dev/sda2 on /oldroot type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sda2 on /oldroot type ext4 (rw,relatime,errors=remount-ro,data=ordered)
udev on /oldroot/dev type devtmpfs (rw,nosuid,relatime,size=16404692k,nr_inodes=4101173,mode=755)
devpts on /oldroot/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /oldroot/dev/shm type tmpfs (rw,nosuid,nodev)
mqueue on /oldroot/dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /oldroot/dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
tmpfs on /oldroot/run type tmpfs (rw,nosuid,noexec,relatime,size=3286976k,mode=755)
tmpfs on /oldroot/run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
cgmfs on /oldroot/run/cgmanager/fs type tmpfs (rw,relatime,size=100k,mode=755)

We mount /proc first since mount relies on it. (You can verify this by running strace mount).

Note : The mount point list was pretty, big so I piped it to head to show the first 10 lines.

Most of these mounts shouldn’t be part of our container so let’s unmount everything and mount /proc again.

root@tupperware:/# umount -a
umount: can't unmount /oldroot/btrfs: Resource busy
umount: can't unmount /oldroot: Resource busy
umount: can't unmount /oldroot: Resource busy
root@tupperware:/# mount -t proc proc /proc
root@tupperware:/# mount
/dev/sda2 on /oldroot type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sda2 on /oldroot type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/loop0 on /oldroot/btrfs type btrfs (ro,relatime,space_cache,subvolid=5,subvol=/)
/dev/loop0 on / type btrfs (ro,relatime,space_cache,subvolid=258,subvol=/containers/tupperware)
proc on /proc type proc (rw,relatime)

umount failed for /oldroot because the resource was busy. This is why /oldroot still shows up when we run mount. Let’s fix that.

root@tupperware:/# umount -l /oldroot
root@tupperware:/# mount
/dev/loop0 on / type btrfs (ro,relatime,space_cache,subvolid=258,subvol=/containers/tupperware)
/dev/loop0 on / type btrfs (ro,relatime,space_cache,subvolid=258,subvol=/containers/tupperware)
proc on /proc type proc (rw,relatime)

The -l flag in the umount command stands for lazy. According to the umount man page, this flag will “detach the filesystem from the file hierarchy now, and clean up all references to this filesystem as soon as it is not busy anymore.” As a result, we don’t see any references to /oldroot when we rerun mount.

Networking

Let’s verify that we don’t have access to the network.

root@tupperware:/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: Network unreachable
root@tupperware:/# ifconfig -a
lo Link encap:Local Loopback
          LOOPBACK MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Good! We don’t have network access, and we only have the loopback interface. Let’s fix that.

Note : We run the following commands from a separate terminal (not in our container).

We need the process id of our container (the PID of unshare).

root@desktop-nicolas:/home/nmesa# CPID=$(pidof unshare)
root@desktop-nicolas:/home/nmesa# echo $CPID
24551

We assign the PID to the CPID variable to make it easier to copy/paste commands if you are following along. Let’s create a pair of virtual network interfaces.

root@desktop-nicolas:/home/nmesa# ip link add name h$CPID type veth peer name c$CPID

The ip link add command creates two interfaces of type veth (virtual ethernet interface). One with name h24551 and the other one with name c24551. h stands for host, c stands for container and 24551 is the PID of our container.

Let’s move the c24551 interface to our container (note this command is also executed from the host).

root@desktop-nicolas:/home/nmesa# ip link set c$CPID netns $CPID

The ip link set command sets configuration attributes to interfaces. In this case, it sets the netns (network namespace) to be the same as the process id stored in $CPID (24551 in our case).

Let’s get back to our container terminal to verify that it got the new interface.

root@tupperware:/# ifconfig -a
c24551 Link encap:Ethernet HWaddr 96:18:F1:61:12:A2
          BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback
          LOOPBACK MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Great! We have the interface inside our container! Let’s get back to our host terminal and attach the h24551 interface to the docker bridge (docker0).

root@desktop-nicolas:/home/nmesa# ip link set h$CPID master docker0 up
root@desktop-nicolas:/home/nmesa# ifconfig docker0
docker0 Link encap:Ethernet HWaddr 02:42:ac:03:c8:91
          inet addr:172.17.0.1 Bcast:172.17.255.255 Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe03:c891/64 Scope:Link
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:77485 errors:0 dropped:0 overruns:0 frame:0
          TX packets:77624 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:6199951 (6.1 MB) TX bytes:164681776 (164.6 MB)

We use the ip link set command to attach the interface to the docker0 bridge. We also grab the ip address of our docker0 interface: 172.17.0.1 (write it down as you’ll need this later).

Note : Jérôme didn’t do this last step, and I suspect this is why the demo didn’t work as expected for him.

Let’s go back to our container terminal and configure the network interface.

root@tupperware:/# ip link set lo up
root@tupperware:/# ip link set c24551 name eth0 up
root@tupperware:/# ip addr add 172.17.42.3/16 dev eth0
root@tupperware:/# ip route add default via 172.17.0.1
root@tupperware:/# ifconfig
eth0 Link encap:Ethernet HWaddr 96:18:F1:61:12:A2
          inet addr:172.17.42.3 Bcast:0.0.0.0 Mask:255.255.0.0
          inet6 addr: fe80::9418:f1ff:fe61:12a2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:35 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3789 (3.7 KiB) TX bytes:936 (936.0 B)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

Let’s break this down:

We bring up the loopback interface.
We bring up the c24551 interface and rename it to eth0. Note that this command can’t be copied and pasted since your interface probably has a different name.
We assign the ip address of 172.17.42.3 to our eth0 interface. Note that this ip address has to be within the range that we got back from the docker0 interface (172.17.0.0/16 in our case).
We set the default gateway of our container to be 172.17.0.1 (this should be the ip address that we got from docker0).
We run a quick ifconfig to make sure everything is setup correctly.

Now, the moment of truth. Let’s ping 8.8.8.8.

root@tupperware:/# ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=11.045 ms
64 bytes from 8.8.8.8: seq=1 ttl=121 time=10.981 ms
64 bytes from 8.8.8.8: seq=2 ttl=121 time=10.508 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 10.508/10.844/11.045 ms

Our container has network connectivity!

Let’s finish the job

As Jérôme noted in the video, we are still running bash inside the container, but the alpine image doesn’t have bash. Up until this point, we’ve been doing the container runtime’s job. To finish the container handoff, we run the following command:

root@tupperware:/# exec chroot / sh
/ #

The exec command is a built-in utility in bash (who knew?). It replaces the shell with the command (chroot in our case) without starting a new process. Once this command is executed, we’re officially in a container. Let’s run a few commands.

/ # apk
apk-tools 2.10.0, compiled for x86_64.

Installing and removing packages:
  add Add PACKAGEs to 'world' and install (or upgrade) them, while ensuring that all dependencies are met
  del Remove PACKAGEs from 'world' and uninstall them

System maintenance:
  fix Repair package or upgrade it without modifying main dependencies
  update Update repository indexes from all remote repositories
  upgrade Upgrade currently installed packages to match repositories
  cache Download missing PACKAGEs to cache and/or delete unneeded files from cache

Querying information about packages:
  info Give detailed information about PACKAGEs or repositories
  list List packages by PATTERN and other criteria
  dot Generate graphviz graphs
  policy Show repository policy for packages

Repository maintenance:
  index Create repository index file from FILEs
  fetch Download PACKAGEs from global repositories to a local directory
  verify Verify package integrity and signature
  manifest Show checksums of package contents

Use apk <command> --help for command-specific help.
Use apk --help --verbose for a full command listing.

This apk has coffee making abilities.
/ # ls
NICK_WAS_HERE dev home media oldroot root sbin sys usr
bin etc lib mnt proc run srv tmp var

Cleanup

Here are some cleanup steps commands.

/ # exit
root@desktop-nicolas:/btrfs# exit
root@desktop-nicolas:/btrfs# cd /
root@desktop-nicolas:/# umount /btrfs
root@desktop-nicolas:/# exit
exit
nmesa@desktop-nicolas:~/demos/containers$ rm -f disk.img
nmesa@desktop-nicolas:~/demos/containers$

Note : I didn’t have to delete my veth interfaces. I’m not sure if they are automatically removed once the network namespace dies.

Conclusion

A lot goes on under the hood to provision containers! This was only covering the namespaces part. There’s also cgroups, capabilities, devices, etc. As Jérôme suggests, don’t create your own container runtime and use docker instead.

Links / Further reading

Var vs Let vs Const

Nicolas Mesa — Sat, 18 Aug 2018 21:06:09 +0000

Cross-posted from my personal blog https://blog.nicolasmesa.co.

Hi there!

In this post, we’re going to talk about javascript. We’ll start by looking at some examples of var, let and const variable declarations and their properties. Then we’re going to go through my recommendations of when to use each one.

var

var is the original way to do variable declaration in javascript. Here’s an example of how you can declare a variable using var:

var myVariable = 10;

var is function-scoped

Variables defined using var are function scoped. This means that the variable is available anywhere within its enclosing function. For example:

function functionScopedExample() {
    var a = 10;

    if (a < 20) {
        var b = 15;
    } else {
        var c = 30;
    }

    console.log(a, b, c); // logs: 10 15 undefined
}

There are a few things happening in this example. First, a is available everywhere in this function (like in most programming languages). Second, b is also available everywhere even though we declared and assigned it inside an if block. Third, c is also available everywhere in the function (even though our execution path doesn’t go into the else statement)! One thing to note here is that the value of c is undefined (since the code in the else statement is not executed), but c was still declared. To understand why this happened, we’ll need to learn about Variable Hoisting.

Here is another example of function-scoped variables:

function functionScopedExample2() {
    var a = 10;

    function innerFunction() {
        var b = 20;
        console.log(a); // logs: 10
    }

    innerFunction();

    console.log(a); // logs: 10
    console.log(b); // error: Uncaught ReferenceError: b is not defined
}

This example shows that a is available in its enclosing function (functionScopedExample2). This includes other functions defined within it (for example the innerFunction function). b, however, can only be referenced from its enclosing function (innerFunction) and not from outside.

Function-scoped declarations can have weird effects:

function functionScopedExample3() {
    var myNumberArray = [];

    for (var i = 0; i < 10; i++) {
        setTimeout(function() {
            myNumberArray.push(i);
        }, 100);

    }

    // wait until all timeouts have executed.
    setTimeout(function() {
        console.log(myNumberArray); // logs: [10, 10, 10, 10, 10, 10, 10, 10, 10, 10]
    }, 500);
}

The example above looks pretty weird. Most programmers would expect myNumberArray to have numbers from 0 - 9. Instead the array contains the number 10 10 times! This happens because the variable i is in scope for the whole function (functionScopedExample3). By the time the functions in the setTimeout execute, i already has a value of 10. One way to fix this issue is to create another function that receives i as an argument and executes the setTimeout:

function fixedFunctionScopedExample3() {
    var myNumberArray = [];

    // a new scope is created every time this function is called.
    // As a result, newI is isolated.
    function executeTimeout(newI) {
        setTimeout(function() {
            myNumberArray.push(newI);
        }, 100);
    }

    for (var i = 0; i < 10; i++) {
        executeTimeout(i);
    }

    // wait until all timeouts have executed.
    setTimeout(function() {
        console.log(myNumberArray); // logs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    }, 500);
}

Later, we’ll see how to solve this problem by using let instead of var.

Variable hoisting

When we define a variable using var, the variable declaration (not the assignment) is hoisted up to the beginning of the function. After the hoisting process, functionScopedExample (defined above) would end up looking like this:

function varHoistingExample() {
    // variables are hoisted to the top and all of them start
    // with the value of undefined.
    var a;
    var b;
    var c;

    // a gets its value here.
    a = 10;

    if (a < 20) {
        // b gets its value here.
        b = 15;
    } else {
        // c never gets its value so it remains undefined.
        c = 30;
    }

    console.log(a, b, c); // logs: 10 15 undefined
}

Hoisting can cause a lot of confusion. Take a look a the following example:

function weirdHoisting() {
    color = 'yellow';
    var color;

    console.log('My favorite color is', color); // logs: My favorite color is yellow
}

Wait, what? How did that work? Remember, first the variables defined with var are hoisted and then the function code is executed. In this example, the code after variable hoisting would look something like this:

function weirdHoisting() {
    // variable declaration is hoisted to the top of its enclosing function.
    var color;
    color = 'yellow';

    console.log('My favorite color is', color); // logs: My favorite color is yellow
}

Variable color’s declaration is hoisted and thus prevents any reference errors.

Named function declarations are hoisted as well (there are some edge-cases if you’re doing conditionals). Here’s an example:

function functionHoisting() {
    hoisted(); // logs: I am hoisted
    notHoisted(); // error: Uncaught TypeError: notHoisted is not a function

    var notHoisted = function() {
        console.log('This will not work');
    }

    function hoisted() {
        console.log('I am hoisted');
    }
}

In this example, we see that the hoisted function can be called before it is declared. The notHoisted variable, however, hasn’t been assigned the function by the time it is called. Note that the function doesn’t fail because the notHoisted name hasn’t been declared, but because it doesn’t point to a function (yet). After the hoisting process, this function would end up looking something like this:

function functionHoisting() {
    // variable and function declaration are hoisted.
    var notHoisted;

    function hoisted() {
        console.log('I am hoisted');
    }

    hoisted(); // logs: I am hoisted
    notHoisted(); // error: Uncaught TypeError: notHoisted is not a function

    // function would be assigned here but we never reach this.
    notHoisted = function() {
        console.log('This will not work');
    }
}

let and const

let is a newer way to define variables in javascript. Here’s an example of a variable definition using let:

let myVariable = 10;

const is a way to define constants in javascript. Here’s an example of a constant defined using const:

const myConstant = 200;

let and const are block-scoped

let and const are both block-scoped. This means that the variable/constant is available anywhere in the block after it has been declared. let/const definitions are NOT hoisted. Let’s go through a few examples:

function blockScopedExample() {
    let a = 10;

    if (a < 20) {
        let b = 15;
    } else {
        let c = 30;
    }

    console.log(a); // logs: 10
    console.log(b); // error: Uncaught ReferenceError: b is not defined

    // we would never reach this since the code fails in the line before
    // but this would be the output if it had been executed
    console.log(c); // error: Uncaught ReferenceError: c is not defined
}

This example shows that b and c are block-scoped. This is the reason why they’re not available outside of the if/else block. a was defined at the beginning of the function, making it available anywhere inside the function block. Declaring something with let at the beginning of a function is analogous to defining it with var.

Remember the example where our array ended up with the same values? Well, this problem is easily solvable by using let instead of var to define i:

function blockScopedExample2() {
    let myNumberArray = [];

    // let makes the variable i block-scoped
    for (let i = 0; i < 10; i++) {
        setTimeout(function() {
            myNumberArray.push(i);
        }, 100);

    }

    // wait until all timeouts have executed.
    setTimeout(function() {
        console.log(myNumberArray); // logs: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    }, 500);
}

Since the variable i is block-scoped instead of function-scoped, its value remains the same for the whole block execution.

Here’s another example showing that variables defined with let are not hoisted:

function noHoisting() {
    color = 'yellow'; // error: Uncaught ReferenceError: color is not defined
    let color;

    // this is never reached.
    console.log('My favorite color is', color);
}

The output of this function is what most programmers coming from other languages expect (having to declare a variable before using it).

So far, every example (except for blockScopedExample2) we’ve covered using let would have the same outcome if we had used const instead. So, what does const give us?

Constants

const is how constants can be declared in javascript. After a constant has been defined using const, the constant can’t have another value assigned. Here’s an example:

function cantReassignConstants() {
    const myConstant = 10;
    myConstant = 9; // error: Uncaught TypeError: Assignment to constant variable
}

Constants defined with const also have to be assigned at declaration time (unlike let and var). For example, the following piece of code would fail during parsing (not at run-time).

function constantsCantBeDeclaredAndNotDefined() {
    const myConstant;
    myConstant = 5;
}

If you try to define this function, you would see an error like this:

Uncaught SyntaxError: Missing initializer in const declaration

Since this error occurs during parsing, the declaration of the function fails and we can’t even call it.

constantsCantBeDeclaredAndNotDefined(); // error: Uncaught ReferenceError: constantsCantBeDeclaredAndNotDefined is not defined

One thing to note about constants defined with const is that only the assignment is constant. The object assigned to a constant doesn’t become immutable. For example, the following code runs with no errors:

function constDoesntMakeObjectsImmutable() {
    const myObjectConstant = {
        myKey: 'myValue'
    };

    myObjectConstant.myKey = 'something else';

    console.log('myObjectConstant.myKey =', myObjectConstant.myKey); // logs: myObjectConstant.myKey = something else
}

Object.freeze()

If you want to make the object immutable, you can use Object.freeze() before assigning the object:

function freezingObject() {
    const myFrozenObject = Object.freeze({
        myKey: 'This value cannot be changed'
    });

    myFrozenObject.myKey = 'something else';

    console.log('myFrozenObject.myKey = ', myFrozenObject.myKey); // logs: myFrozenObject.myKey = This value cannot be changed
}

Even though the value of myKey didn’t change, the assignment didn’t fail either (it failed silently). If you want it to fail, use the 'use strict;' at the beginning of the function like this:

function freezingObject() {
    'use strict';
    const myFrozenObject = Object.freeze({
        myKey: 'This value cannot be changed'
    });

    myFrozenObject.myKey = 'something else'; // error: Uncaught TypeError: Cannot assign to read only property 'myKey' of object '#<Object>'

    // never executed
    console.log('myFrozenObject.myKey = ', myFrozenObject.myKey);
}

Object.freeze will only freeze the object that we pass as an argument. If the object contains references to other objects, those can still be modified unless we freeze them as well. For example:

function objectFreezeIsShallow() {
    const myFrozenObject = Object.freeze({
        key1: {
            reassignable: 'reassign me'
        },
        key2: Object.freeze({
            notReassignable: 'This value cannot change'
        })
    });

    myFrozenObject.key1.reassignable = 'reassigned';
    myFrozenObject.key2.notReassignable = 'this will not work';

    console.log('myFrozenObject.key1.reassignable =', myFrozenObject.key1.reassignable); // logs: myFrozenObject.key1.reassignable = reassigned
    console.log('myFrozenObject.key2.notReassignable =', myFrozenObject.key2.notReassignable); // logs: myFrozenObject.key2.notReassignable = This value cannot change
}

In this case, neither key1 or key2 can be reassigned since they’re frozen. The inner object of key1 however, is not frozen and the reassignable key is set to a different value. Note that the notReassignable key didn’t have its value reassigned since the object was frozen.

Recommendations

So, which one(s) should we use and why?

There are lots of reasons to use one over the other, so I divide this into a section per reason. Note that these are my preferences and it’s perfectly fine for other programmers to think differently :)

Never use the three of them in the same project

Decide which ones you want to use and stick to those. Using all three of them will be confusing. It will be difficult to know which one to pick and the code will be harder to understand. Be nice to the future readers of your code (including yourself).

Prefer consistency

If you’re working on an old project that is using var, keep using var. This will keep the project consistent.

Put variable declarations at the top if using var

If you’re using var, place all variable declarations at the top of the function. This will help you see which variables are available in the function and there won’t be any hoisting surprises.

Consider older browser support

If your code needs to support older browsers that don’t have let or const, consider using var instead. Alternatively, use a transpiler that converts all let and const declarations to var.

Prefer let over var

If you’re starting a new project, I would encourage you to use let over var. let is more predictable, especially for developers coming from a different language.

Prefer const over let when possible

I am a firm believer that everything should be a constant except for very special cases in loops or things like that (in those cases, fall-back to let). I’ll probably write another post on why using const (final in Java) is a good choice, but the gist is:

Debugging

It will ease debugging. For instance, consider the following code:

function useConstWhenPossible(myOtherObject) {
    const myObject = myOtherObject.getMyObject();

    // ...
    // 100 lines of code
    // ...

    myObject.doSomething(); // error: Uncaught TypeError: Cannot read property 'doSomething' of undefined

    // ...
    // 100 lines of code
    // ...

    return ...;
}

When you see the error, you go to the constant declaration. You can immediately tell that myOtherObject.getMyObject() returned undefined. There is no need to look through the other 100 lines of code to find if myObject ever gets reassigned.

Variable naming

You will have to come up with good names for your constants because you won’t be able to reuse them. This will make your code easier to read and more self-documenting.

Helps prevent bugs

Another developer (or maybe even you) won’t have a chance to define an existing variable and break the whole function.

Use named functions

This will save you a lot of headaches, especially when moving code around. You won’t need to make sure that that your function calls are after the function declarations and everything will work consistently.

What Happens When You Type a Url in Your Browser and Press Enter

Nicolas Mesa — Fri, 10 Aug 2018 05:24:55 +0000

Cross-posted from my personal blog https://blog.nicolasmesa.co.

Disclaimer: Hello! This is my first blog post! I’m super excited to share it with the world but should also warn you that it’s a bit long and repetitive. Hopefully, I’ll become a better writer as I blog more. I hope you enjoy it :)

Introduction

I’ve heard this question many times during my career and want to try to answer it. I’m going to dive deep into the networking part of the question since this is the part that I find the most interesting.

A lot of stuff happens under the hood in the network to get an HTTP request from the browser to the server and then an HTTP response from the server back to the browser. To be able to explain what happens, I’m going to make some assumptions about the network setup and the status of the network.

Assumptions

Network status

Our computer is connected to the network via Ethernet.
Our computer already has an IP address (more on what the IP address is later).
Our computer sits behind a NAT (this is the case in many home networks).
Our ARP cache is clear.
Our DNS cache is clear.
Our router’s ARP cache contains an entry for its default gateway.
Our network is 192.168.0.0/24.
The URL we’re going to use is http://www.example.com.
The DNS server’s cache has an entry for www.example.com.
We’re going to ignore TCP sequence numbers.

Configuration

Our computer (computer making the HTTP request)

IP Address: 192.168.0.10.
Default gateway: 192.168.0.1.
DNS Server: 8.8.8.8.
MAC Address: AA:AA:AA:AA:AA:AA.
TCP source port: 9999.
DNS source port: 3333.

Our router / NAT

Internal IP Address: 192.168.0.1.
External IP address: 51.0.0.20.
External TCP source port: 15000.
External DNS source port: 10000.
External net mask: 255.255.255.0 (or /24).
Internal MAC address: BB:BB:BB:BB:BB:BB.
External MAC address: CC:CC:CC:CC:CC:CC.
Default gateway: 51.0.0.1.
Default gateway’s MAC address: DD:DD:DD:DD:DD:DD.

www.example.com server

The IP address of www.example.com is 93.184.216.34.

Step by step

To get this process started we type the URL http://www.example.com in the web browser and press enter. This will instruct our browser to create an HTTP request to send to www.example.com. The HTTP request will look something like this:

GET / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

HTTP request to www.example.com.

Our computer will now try to connect to www.example.com in TCP port 80 (the default HTTP port). First, it needs to create a TCP SYN segment to initiate the TCP connection. The SYN segment contains information to establish the connection including a random source port (9999 from our assumptions), destination port (80) and a flag (SYN).

This TCP segment is then encapsulated in an IP packet that contains the source IP (192.168.0.10) and the destination IP. For the destination IP, our computer goes to its DNS cache and looks for an entry containing www.example.com. Since its cache is empty, the creation of the IP packet needs to wait until we can get the IP address of www.example.com. So far our incomplete IP packet looks something like this:

IP[
    SourceIp=192.168.0.10,
    DestIp=????,
    TCP[
        SourcePort=9999,
        DestPort=80,
        Flags=[SYN]
    ]
]

Incomplete SYN IP packet (note the ???? in the DestIP field).

To get the IP address of www.example.com, we need to use a service called DNS. DNS stands for Domain Name System and is the system responsible for translating human-readable names (such as www.example.com) into IP addresses. Our computer creates a DNS request that includes the type of request (A) and the domain name to resolve (www.example.com). This DNS request is then encapsulated in a UDP datagram whose header contains a random source port (3333 from our assumptions) and the destination port (53 is DNS’s well-known port). This datagram is then placed inside an IP packet whose header has the source IP address (192.168.0.10) and the destination IP address (8.8.8.8). Note that the DNS server was configured beforehand in our computer so we already know what its IP address is.

We now want to send this IP packet but our computer doesn’t know where to send it. The computer looks at its route table to see which route matches the 8.8.8.8 IP address. The route table looks something like this:

Destination Netmask Gateway Interface
0.0.0.0 0.0.0.0 192.168.0.1 eth0
192.168.0.0 255.255.255.0 0.0.0.0 eth0

Route table of our computer.

Our computer uses the longest prefix match to select who to send the packet to. In this case, it selects the first route which is our default gateway (192.168.0.1). This means that the IP packet will be sent to our router and the router will take care of forwarding it to the next hop (which is hopefully closer to the DNS server). The routers in the path will keep forwarding the packet based on their routing tables until it reaches the DNS server (8.8.8.8). This process happens every time our computer needs to send an IP packet and, in our case, it will always choose to send the packet to our default gateway (192.168.0.1).

Now that the IP packet is ready, it is wrapped in an Ethernet frame. The Ethernet header contains the source MAC address (AA:AA:AA:AA:AA:AA) and the destination MAC address (this would be the router’s MAC address in this case). To get the router’s MAC address, our computer looks in its ARP table to see if it has an entry to convert the router’s IP address (192.168.0.1) to the router’s MAC address. Since the ARP cache is empty, we need to find out what is the MAC address before we can send the IP packet. This is how our incomplete Ethernet frame looks like:

Ethernet[
    SourceMac=AA:AA:AA:AA:AA:AA,
    DestMac=????,
    IP[
        SourceIp=192.168.0.10,
        DestIp=8.8.8.8,
        UDP[
            SourcePort=3333,
            DestPort=53,
            DNS[
                Query=A,
                Domain=www.example.com
            ]
        ]
    ]
]

Incomplete Ethernet frame for the DNS query (note the ???? in the DestMac field).

To convert IP addresses to MAC addresses, the ARP protocol is used. ARP stands for Address Resolution Protocol. The ARP packet contains a question asking who-has 192.168.0.1 tell 192.168.0.10 along with the source MAC address (AA:AA:AA:AA:AA:AA), the destination MAC address (FF:FF:FF:FF:FF:FF which is the broadcast address), the source IP address (192.168.0.10), and the destination IP (192.168.0.1 which is the IP we’re asking about). The ARP packet is then wrapped in an Ethernet frame containing the source MAC address (AA:AA:AA:AA:AA:AA) and for the destination MAC address, the broadcast address is used (FF:FF:FF:FF:FF:FF). The frame is sent over our private network and looks something like this:

Ethernet[
    SourceMac=AA:AA:AA:AA:AA:AA,
    DestMac=FF:FF:FF:FF:FF:FF,
    ARP[
        Type=Request,
        SourceIp=192.168.0.10,
        DestIp=192.168.0.1,
        SourceMac=AA:AA:AA:AA:AA:AA,
        DestMac=FF:FF:FF:FF:FF:FF
    ]
]

ARP packet to find out our router’s MAC address.

There is something worth noting at this point. This is the first interaction that our computer has had with the network!

Since the broadcast MAC address (FF:FF:FF:FF:FF:FF) was used, this frame will reach every node on our private network. Every node (except for our router) will end up discarding the packet since the destination IP address (192.168.0.1) is not their IP.

When our router receives this packet, it will see that 192.168.0.10 is asking for its MAC address. It can also see in the ARP packet the source MAC address of our computer. The router saves this information in its ARP table (192.168.0.10 -> AA:AA:AA:AA:AA:AA) and creates an ARP reply saying 192.168.0.1 is-at BB:BB:BB:BB:BB:BB. This ARP reply is then wrapped in an Ethernet frame with the source MAC address (BB:BB:BB:BB:BB:BB) and destination MAC address (AA:AA:AA:AA:AA:AA). The Ethernet frame is now sent over the network. One thing to note here is that the destination MAC address is no longer the broadcast MAC address so only our computer will process it. From now on, our router will be able to convert our computer’s IP address (192.168.0.10) to its MAC address (AA:AA:AA:AA:AA:AA) by referring to its ARP cache. The Ethernet frame containing this reply looks something like this:

Ethernet[
    SourceMac=BB:BB:BB:BB:BB:BB,
    DestMac=AA:AA:AA:AA:AA:AA,
    ARP[
        Type=Reply,
        SourceIp=192.168.0.1,
        DestIp=192.168.0.10,
        SourceMac=BB:BB:BB:BB:BB:BB,
        DestMac=AA:AA:AA:AA:AA:AA
    ]
]

ARP reply where our router tells its MAC address to our computer.

Our computer receives the Ethernet frame, sees the ARP reply and saves the MAC address of our router in its ARP table (192.168.0.1 -> BB:BB:BB:BB:BB:BB). From now on, any time our computer needs to convert our router’s IP address (192.168.0.1) to its MAC address (BB:BB:BB:BB:BB:BB), it simply needs to refer to its ARP cache. We’re making progress! We now have the MAC address of our router, which we need to send the IP packet with the DNS query to get the IP of www.example.com, which we need to send a TCP SYN packet, to establish a TCP connection over port 80, which we need to send the HTTP request.

Since our computer now knows the MAC address of the router, it fills in the blanks of the Ethernet frame containing our DNS query and sends it over the network to (BB:BB:BB:BB:BB:BB). This happens with every Ethernet frame that needs to be sent to our router. The frame looks something like this:

Ethernet[
    SourceMac=AA:AA:AA:AA:AA:AA,
    DestMac=BB:BB:BB:BB:BB:BB,
    IP[
        SourceIp=192.168.0.10,
        DestIp=8.8.8.8,
        UDP[
            SourcePort=3333,
            DestPort=53,
            DNS[
                Query=A,
                Domain=www.example.com
            ]
        ]
    ]
]

Complete Ethernet frame for the DNS query.

One thing to note is that the DestIp field is not the router’s IP address; it’s still 8.8.8.8.

The router receives this frame and looks at the destination IP (8.8.8.8) in the IP header. It also uses the longest prefix match to select where to forward this packet and ends up selecting its default Gateway (51.0.0.1). This decision is made every time the router needs to forward a packet to an IP address in the public Internet such as 8.8.8.8 (and later 93.184.216.34).

Based on our assumptions our router is also a NAT. This means that our router needs to translate the packet before it can be forwarded. NAT stands for Network Address Translation. It is widely deployed to have multiple devices share a single public IP address. Our internal (or private) network uses private IP addresses (192.168.0.1 and 192.168.0.10) but the public Internet uses public IP addresses. When we need to send a packet to the public Internet, the NAT takes care of replacing the private IP address with the public IP address (51.0.0.20). The NAT uses 5 values to identify a connection: protocol (UDP / TCP), source IP, internal source port, destination IP and external source port. The external source port is chosen randomly to make sure the connection identifier is unique. Based on our assumptions, the router decides to use port 10000. The entry in the NAT table looks something like this:

UDP 192.168.0.10 3333 -> 8.8.8.8 10000

NAT table. Note that the destination port (53) is not used to identify the connection.

Our router uses the information in the NAT table to replace the source IP address of the packet (192.168.0.10) with the public IP address (51.0.0.20), and the internal source port (3333) with the external source port (10000). This makes that IP packet routable in the public Internet. Our router performs this translation for any packet that needs to go out to the public Internet including TCP/IP packets. We’ll call this process internal-to-external translation.

Our router wraps the packet in a new Ethernet frame with the source MAC address of CC:CC:CC:CC:CC:CC, and a destination MAC address of DD:DD:DD:DD:DD:DD (we assumed this value was already cached so there’s no need for ARP). Every packet that our router forwards to the public Internet goes through the same process so we won’t bring this up again. The Ethernet frame ends up looking something like this:

Ethernet[
    SourceMac=CC:CC:CC:CC:CC:CC,
    DestMac=DD:DD:DD:DD:DD:DD,
    IP[
        SourceIp=51.0.0.20,
        DestIp=8.8.8.8,
        UDP[
            SourcePort=10000,
            DestPort=53,
            DNS[
                Query=A,
                Domain=www.example.com
            ]
        ]
    ]
]

Publicly routable packet with DNS query (note the change in SourceIp and SourcePort).

The Ethernet frame is sent to the default gateway which is most likely a router owned by our ISP and it is up to the ISP to get our packet to 8.8.8.8.

Once our packet reaches 8.8.8.8 (Google’s DNS server), the server takes a look at the query (A for www.example.com) and searches its cache. We’re assuming that the cache entry exists, so it knows that www.example.com maps to 93.184.216.34. The DNS server creates a new DNS response message containing this information as the answer to the query. The response is wrapped in a UDP datagram with source port 53 and destination port 10000. The UDP datagram is then wrapped in an IP packet with source IP 8.8.8.8 and destination IP 51.0.0.20 (our public IP). The packet is finally wrapped in an Ethernet frame (we’ll ignore this part) and sent through the network. The IP packet ends up looking like this:

IP[
    SourceIp=8.8.8.8,
    DestIp=51.0.0.20,
    UDP[
        SourcePort=53,
        DestPort=10000,
        DNS[
            Answer=93.184.216.34,
            Domain=www.example.com
        ]
    ]
]

IP packet containing the DNS reply.

The packet traverses the network and eventually reaches our router. Our router needs to translate this packet before it forwards it into our private network. The router looks at the protocol (UDP), source IP (8.8.8.8) and the destination port (10000) to find an entry in the NAT table with those values. With the values from the table, the router translates the destination IP to be our computer’s IP (192.168.0.10) and the destination port to be our computer’s source port (3333). This makes that IP packet routable in our private network. The same process occurs for any packet coming from the public Internet including TCP/IP packets (only packets whose properties can be found in the NAT table are translated, the others are dropped). We’ll call this process external-to-internal translation. After this, the router wraps the packet in an Ethernet frame with source MAC BB:BB:BB:BB:BB:BB and destination MAC AA:AA:AA:AA:AA:AA and sends it over the private network (this happens every time our router forwards a packet to our computer so we won’t bring it up again). The Ethernet frame with the translated IP packet looks something like this:

Ethernet[
    SourceMac=BB:BB:BB:BB:BB:BB,
    DestMac=AA:AA:AA:AA:AA:AA,
    IP[
        SourceIp=8.8.8.8,
        DestIp=192.168.0.10,
        UDP[
            SourcePort=53,
            DestPort=3333,
            DNS[
                Answer=93.184.216.34,
                Domain=www.example.com
            ]
        ]
    ]
]

Privately routable packet containing the DNS reply (note the change in DestIp and DestPort).

With the IP address of www.example.com, our computer fills in the destination IP in our incomplete SYN packet. The IP header is wrapped in an Ethernet frame and sent through the private network. The Ethernet frame looks something like this:

Ethernet[
    SourceMac=AA:AA:AA:AA:AA:AA,
    DestMac=BB:BB:BB:BB:BB:BB,
    IP[
        SourceIp=192.168.0.10,
        DestIp=93.184.216.34,
        TCP[
            SourcePort=9999,
            DestPort=80,
            Flags=[SYN]
        ]
    ]
]

Privately routable packet containing the TCP SYN packet.

Our router receives the packet and extracts the protocol (TCP), the source IP (192.168.0.10), the source port (9999), the destination IP (93.184.216.34) and generates a random number to be used as the external source port (15000 from the assumptions). These values identify the TCP connection and are saved in the router’s NAT table. Our router can reference the NAT table to identify packets that need to be translated and what values to use for the translation. The entry in the NAT table looks something like this:

TCP 192.168.0.10 9999 -> 93.184.216.34 15000

Our router uses our public IP address (51.0.0.20) to replace the source IP and the port generated in the previous step (15000) to replace the source port. It then wraps the translated IP packet in an Ethernet frame and sends it over the network. The Ethernet frame looks something like this:

Ethernet[
    SourceMac=CC:CC:CC:CC:CC:CC,
    DestMac=DD:DD:DD:DD:DD:DD,
    IP[
        SourceIp=51.0.0.20,
        DestIp=93.184.216.34,
        TCP[
            SourcePort=15000,
            DestPort=80,
            Flags=[SYN]
        ]
    ]
]

Publicly routable SYN segment (note the changes to SourceIp and SourcePort).

When the packet reaches 93.184.216.34 (the IP address of www.example.com), the server looks at the destination port of the TCP header (80) and checks if an application is listening on that port. Since there is an HTTP server running on that port, the server needs to reply with a SYN/ACK. The server creates a TCP segment and sets its source port to 80, the destination port to 15000 (the source port of the SYN packet that was just received), and SYN/ACK for the flags. It then wraps the segment in an IP packet with source IP of 93.184.216.34 and destination IP of 51.0.0.20. Finally, it wraps the IP packet in an Ethernet frame (we’re going to ignore this part). The packet is then sent out to the network and looks something like this:

IP[
    SourceIp=93.184.216.34,
    DestIp=51.0.0.20,
    TCP[
        SourcePort=80,
        DestPort=15000,
        Flags=[SYN, ACK]
    ]
]

IP packet containing the TCP SYN / ACK.

When the packet reaches our router, it performs an external-to-internal translation. The translated packet is wrapped in an Ethernet frame and sent over the private network. The Ethernet frame looks something like this:

Ethernet[
    SourceMac=BB:BB:BB:BB:BB:BB,
    DestMac=AA:AA:AA:AA:AA:AA,
    IP[
        SourceIp=93.184.216.34,
        DestIp=192.168.0.10,
        TCP[
            SourcePort=80,
            DestPort=9999,
            Flags=[SYN, ACK]
        ]
    ]
]

Privately routable packet containing the TCP SYN / ACK.

Our computer looks at the destination port (9999) and the TCP flags (SYN/ACK) and checks the state of this connection. The only thing missing to establish this connection is to send an ACK back to the server (93.184.216.34). Our computer creates a new TCP segment with source port 9999, destination port 80 and ACK as its only flag. This segment is then wrapped in an IP packet with source IP 192.168.0.10 and destination IP 93.184.216.34. The IP packet is wrapped in an Ethernet frame and sent over the private network. The Ethernet frame looks something like this:

Ethernet[
    SourceMac=AA:AA:AA:AA:AA:AA,
    DestMac=BB:BB:BB:BB:BB:BB,
    IP[
        SourceIp=192.168.0.10,
        DestIp=93.184.216.34,
        TCP[
            SourcePort=9999,
            DestPort=80,
            Flags=[ACK]
        ]
    ]
]

Privately routable packet containing the ACK to complete the three-way handshake.

Our router receives this frame and uses the information in the NAT table to perform an internal-to-external translation. Then, it wraps the translated IP packet in an Ethernet frame and sends it over the public network. The Ethernet frame looks something like this:

Ethernet[
    SourceMac=CC:CC:CC:CC:CC:CC,
    DestMac=DD:DD:DD:DD:DD:DD,
    IP[
        SourceIp=50.0.0.20,
        DestIp=93.184.216.34,
        TCP[
            SourcePort=15000,
            DestPort=80,
            Flags=[ACK]
        ]
    ]
]

Publicly routable packet containing the ACK to complete the three-way handshake.

The packet traverses the Internet until it gets to 93.184.216.34. The server uses the source port (15000) and the source IP (50.0.0.20) to check the connection state and marks the connection as ESTABLISHED. Note that the server doesn’t return an ACK for the ACK it just received.

Our computer is now ready to send the actual HTTP request. It wraps the HTTP request in a TCP segment with source port 9999, destination port 80 and sets the flags to ACK/PUSH. This TCP segment is then wrapped in an IP packet with source IP 192.168.0.10 and destination IP 93.184.216.34. Then, it wraps the IP packet in an Ethernet frame and sends it through the private network. The frame looks something like this:

Ethernet[
    SourceMac=AA:AA:AA:AA:AA:AA,
    DestMac=BB:BB:BB:BB:BB:BB,
    IP[
        SourceIp=192.168.0.10,
        DestIp=93.184.216.34,
        TCP[
            SourcePort=9999,
            DestPort=80,
            Flags=[ACK, PUSH],
            HTTP[
                GET / HTTP/1.1
                Host: www.example.com
                ...
            ]
        ]
    ]
]

Privately routable packet containing the HTTP request.

Our router receives the frame and does an internal-to-external translation to the IP packet. It then wraps the translated packet in an Ethernet frame and sends it over the public network. The Ethernet frame looks something like this:

Ethernet[
    SourceMac=CC:CC:CC:CC:CC:CC,
    DestMac=DD:DD:DD:DD:DD:DD,
    IP[
        SourceIp=50.0.0.20,
        DestIp=93.184.216.34,
        TCP[
            SourcePort=15000,
            DestPort=80,
            Flags=[ACK, PUSH],
            HTTP[
                GET / HTTP/1.1
                Host: www.example.com
                ...
            ]
        ]
    ]
]

Publicly routable packet containing HTTP request.

The packet reaches 93.184.216.3 (www.example.com) and the server uses the source IP (51.0.0.20) and source port (15000) to verify that the connection exists and is ESTABLISHED. The PUSH flag in the TCP segment instructs the server to send the body of the TCP segment (the HTTP request) to the application listening on port 80 (the HTTP server). The HTTP server processes the HTTP request and generates an HTTP response. An HTTP response from www.example.com looks something like this:

HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: max-age=604800
Content-Type: text/html
Date: Sat, 11 Aug 2018 20:45:46 GMT
Etag: "1541025663"
Expires: Sat, 18 Aug 2018 20:45:46 GMT
Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
Server: ECS (oxr/830D)
Vary: Accept-Encoding
X-Cache: HIT
Content-Length: 606

<!doctype html>
<html>
...
</html>

Sample HTTP response from www.example.com (snipped almost all of the body).

Note: The server should have sent and ACK to acknowledge that this TCP segment was received. However, this ACK can also be sent along with the HTTP response, so in the interest of not being even more repetitive, we’ll assume this is the case.

The HTTP response is wrapped in a TCP segment with source port 80, destination port 15000 and flags ACK/PUSH. Then, the TCP segment is wrapped in an IP packet with source IP 93.184.216.34 and destination IP 50.0.0.20. The packet is then wrapped in an Ethernet frame (that we are also going to ignore) and sent through the public network. The IP packet looks something like:

IP[
    SourceIp=93.184.216.34,
    DestIp=50.0.0.20,
    TCP[
        SourcePort=80,
        DestPort=15000,
        Flags=[ACK, PUSH],
        HTTP[
            HTTP/1.1 200 OK
            Accept-Ranges: bytes
            ...
            <!doctype html>
            ...
        ]
    ]
]

IP packet containing the HTTP response.

Our router receives the packet and performs an external-to-internal translation. After that, the IP packet is wrapped with an Ethernet frame and sent over the private network. The Ethernet frame looks something like this:

Ethernet[
    SourceMac=BB:BB:BB:BB:BB:BB,
    DestMac=AA:AA:AA:AA:AA:AA,
    IP[
        SourceIp=93.184.216.34,
        DestIp=192.168.0.10,
        TCP[
            SourcePort=80,
            DestPort=9999,
            Flags=[ACK, PUSH],
            HTTP[
                HTTP/1.1 200 OK
                Accept-Ranges: bytes
                ...
                <!doctype html>
                ...
            ]
        ]
    ]
]

Privately routable packet with the HTTP response.

Our computer receives the packet and looks up the connection information. The PUSH flag instructs our computer to send the TCP body (the HTTP response) to the application that opened the connection (our web browser). Our web browser now has the HTTP response and can use it to render the web page. From the networking side, however, we’re not done. The server doesn’t know that we received the TCP segment since we haven’t acknowledged it (sent an ACK back). Since the browser is not going to send more requests to www.example.com, it will also close the connection. These two actions can be combined into one by sending the ACK and the FIN flags in the same TCP segment (we’re going to assume this happened to avoid even more repetition).

Our computer creates a new TCP segment with source port 9999, destination port 80 and FIN/ACK as its flags. It places this segment in an IP packet with source IP 192.168.0.10 and destination IP 93.184.216.34. The IP packet is wrapped with an Ethernet frame and is sent over the network. The Ethernet frame looks something like this:

Ethernet[
    SourceMac=AA:AA:AA:AA:AA:AA,
    DestMac=BB:BB:BB:BB:BB:BB,
    IP[
        SourceIp=192.168.0.10,
        DestIp=93.184.216.34,
        TCP[
            SourcePort=9999,
            DestPort=80,
            Flags=[ACK, FIN]
        ]
    ]
]

Privately routable packet containing the ACK for the HTTP response and the FIN to close the TCP connection.

The router gets this frame and performs an internal-to-external translation. Then, it wraps the IP packet in an Ethernet frame and sends the frame to the public network. The frame looks something like this:

Ethernet[
    SourceMac=CC:CC:CC:CC:CC:CC,
    DestMac=DD:DD:DD:DD:DD:DD,
    IP[
        SourceIp=50.0.0.20,
        DestIp=93.184.216.34,
        TCP[
            SourcePort=15000,
            DestPort=80,
            Flags=[ACK, FIN]
        ]
    ]
]

Publicly routable packet with the ACK for the HTTP response and the FIN to close the TCP connection.

When the server receives this packet, it now knows that the HTTP response was received. From the flags, it can also tell that our computer wants to close the connection. The server creates a new TCP segment with source port 80, destination port 15000 and flags FIN/ACK. It wraps this TCP segment in an IP packet with source IP 93.184.216.34 and destination IP 50.0.0.20. The IP packet is sent over the network. The IP packet looks something like this:

IP[
    SourceIp=93.184.216.34,
    DestIp=50.0.0.20,
    TCP[
        SourcePort=80,
        DestPort=15000,
        Flags=[ACK, FIN]
    ]
]

IP packet containing the FIN to close the connection.

When our router receives the packet, it performs an external-to-internal translation and then wraps the packet in an Ethernet frame. It then sends the frame through our private network. The frame looks something like this:

Ethernet[
    SourceMac=BB:BB:BB:BB:BB:BB,
    DestMac=AA:AA:AA:AA:AA:AA,
    IP[
        SourceIp=93.184.216.34,
        DestIp=192.168.0.10,
        TCP[
            SourcePort=80,
            DestPort=9999,
            Flags=[ACK, FIN]
        ]
    ]
]

Privately routable packet containing the FIN to close the connection.

Our computer receives this packet and finds a FIN in the flags. The connection is now closed, all that remains is for our computer to send an acknowledgement to the server that it received the FIN/ACK packet. It creates a TCP segment with source port 9999 and destination port 80. The segment is wrapped in an IP packet with source IP 192.168.0.10 and destination IP 93.184.216.34. The IP packet is wrapped in an Ethernet frame and sent over the private network. The frame looks something like this:

Ethernet[
    SourceMac=AA:AA:AA:AA:AA:AA,
    DestMac=BB:BB:BB:BB:BB:BB,
    IP[
        SourceIp=192.168.0.10,
        DestIp=93.184.216.34,
        TCP[
            SourcePort=9999,
            DestPort=80,
            Flags=[ACK]
        ]
    ]
]

Privately routable packet containing last ACK to acknowledge that the connection has been closed.

The router receives this frame and performs an internal-to-external translation. The IP packet is wrapped in an Ethernet frame sent over the public network. The Ethernet frame looks something like this:

Ethernet[
    SourceMac=CC:CC:CC:CC:CC:CC,
    DestMac=DD:DD:DD:DD:DD:DD,
    IP[
        SourceIp=50.0.0.20,
        DestIp=93.184.216.34,
        TCP[
            SourcePort=15000,
            DestPort=80,
            Flags=[ACK]
        ]
    ]
]

Publicly routable packet with translated packet with last ACK.

When the server gets this packet, it looks at the flags and confirms that our computer got the FIN TCP segment.

Conclusion

Wow, that was a long post! I think we can all agree that a lot happens at the network layer from the time you type a URL in the browser and press enter until you get to see your web page. There are a few important things to call out though:

This was just one HTTP request! Usually, a page that has external scripts, images, stylesheets, etc, makes tens or hundreds of requests!
With only one request, we were able to see the value of caching. We only had to ask for the MAC address of the router once and then we were able to fetch that MAC address from the cache. If we had made more requests, DNS caching would have also played a role since we would have cached the IP address of www.example.com and wouldn’t have had to ask our DNS server for it.
Caching plays a big role in HTTP as well. If our browser had cached this page before, we wouldn’t have had to make any network calls!
There is some overhead while establishing the TCP connection. HTTP clients try to keep the connection alive (with the Connection: Keep-Alive HTTP header) to avoid this overhead and reuse the same connection. A problem with this header is that HTTP clients can only make one request a time (per TCP connection). HTTP/2 and QUIC solve this problem by multiplexing the connection.

Thanks for reading! I hope you enjoyed it and learned something!

About

Nicolas Mesa — Tue, 07 Aug 2018 06:17:59 +0000

Hi and welcome to my blog!

I am a Software Development Engineer working for Amazon on the Amazon Go and Amazon Books team. Currently, I live in Seattle with my wife, Elisa, and my daughter, Amalia. My interests are in Web technologies, Javascript, Python, Networking, AWS, Docker, and Linux.

Before joining Amazon, I was in New York City pursuing my Master’s degree in Computer Science at Columbia University. Prior to that, I worked at Alegra, a startup in Colombia to help small and medium-sized companies with their accounting and bookkeeping.

I am also a water skier and did it competitively for a while.

The purpose of this blog is to publish my notes about stuff that I find interesting or about something new that I learned. Hopefully, I’ll look back in a few years and will see how much I’ve learned.

DEV Community: Nicolas Mesa

Booting Your Own Kernel on Raspberry Pi via Uart

Requirements

Prerequisites

Setup

Git tags

screen replacement

Constructor

Utility methods

start_interactive

Sending the kernel over UART

Protocol

boot_send modifications

UartConnection

Kernel checksum

send_kernel

Calling the send_kernel function

Kernel side modifications

branch_to_address

uart_send_int / uart_read_int

include utils.h

readline

strcmp

copy_and_jump_to_kernel

kernel_main

Testing our UART boot

Build the kernel and copy it to the SD card

Booting test: sending the same kernel

Booting test: making a small change

Booting test: adding a function

The fix

Options

Option 1: Copy the kernel to a different location and jump there

Option 2: Copy the current running kernel to a new address range, jump to it, copy the new kernel to the original address range, jump back to it

Fix implementation

Position Independent Code

Determining current kernel size

copy_current_kernel_and_jump

Call copy_current_kernel_and_jump from kernel_main

Testing the fix

Testing the previous error

Sending the same kernel multiple times

Conclusion

Improvements

Links / References

SaaS Like Isolation in Django Rest Framework

Use-cases

Start the project

The accounts app

The User Messages App

Refactoring the serializer

Refactoring the views

Preventing user impersonation & protecting user privacy

Testing it out

Further improvements

Conclusion

Links

How to Access Logged Objects in the Javascript Console

TLDR

Explanation

The Hack Version

The better version

Conclusion

How Does Traceroute Work

Traceroute example

IP Header

Traceroute introduction

Capturing the traceroute traffic

Traceroute traffic deep dive

Conclusion

Links

Serve Your Current Directory With Python and HTTP

TLDR

Explanation

Setup

Find out your python version

Python 3 command

Python 2 command

Test it out

Root directory

`screen` replacement

Calling the `send_kernel` function

Call `copy_current_kernel_and_jump` from `kernel_main`