DEV Community: Simon Richard

MongoDB Hackathon Submission - QrTracker

Simon Richard — Thu, 13 Jan 2022 23:56:32 +0000

Overview of My Submission

QrTracker is a multi-purpose tracking alternative that gives small business owners (or anyone else, really) the ability to create, as well as consume, tracking information.

It's similar to the software that companies like FedEx and UPS use to give their customers tracking information, but it's open source.

QrTracker also allows for more freedom in how it's used; it supports, but does not require, geographical information.

Submission Category:

Action Star

Link to Code

jsimonrichard / QrTracker-atlas-hackathon

Submission for the Atlas Hackathon put on by MongoDB and dev.to

Screenshots

MongoDB Technologies Used

Realm Functions
Realm Triggers
Atlas Search
GraphQL Integration

Other Technologies Used

Git / GitHub
GraphQL
ReactJS
BlueprintJS
Courier
mailersend

Creating a Custom Ubuntu Server Disk Image for the Raspberry Pi with CustomPiOS

Simon Richard — Thu, 16 Dec 2021 21:59:17 +0000

This article will walk through the creation of a custom disk image for the Raspberry Pi. Possible use cases include...

Making a Raspberry Pi disk image with all of your favorite apps pre-installed
Making a distributable file for a product that runs on the Raspberry Pi (something like OctoPrint)

All of this is made possible by CustomPiOS, a tool that opens, modifies, and repackages pre-existing ARM images (usually a Raspbian / Raspberry Pi OS image downloaded from the official website).

This particular article will focus on making a 64-bit Ubuntu Server spin-off for the Raspberry Pi that runs docker-compose on boot. The process for making a Raspberry Pi OS spin-off is similar, but I would like to focus on the Ubuntu Server because it poses a few unique challenges that are worth covering.

Getting Started

First, let's clone the CustomPiOS github repo and cd into the src folder.

git clone https://github.com/guysoft/CustomPiOS
cd CustomPiOS/src

We'll also want to install these dependancies:

sudo apt install qemu-user-static p7zip-full

Depending on your machine, you may need some other dependencies as well. A full list can be found on the CustomPiOS github page.

Next, we need to run the make_custom_pi_os script. This will create a folder for our custom distro's config files.

./make_custom_pi_os custom_distro

If you're creating a spin-off of Raspberry Pi OS, you may also want to add the -g flag. This will automatically download the latest version of Rasbian / Raspberry Pi OS.

Next, let's change directory into the folder we just created

cd custom_distro/src

This is the directory we'll be working in from now on. It's structure should be as follows.

.
├── build_dist
├── config
├── custompios_path
├── image
│   └── README
├── modules
│   └── custom_server
│       ├── config
│       ├── end_chroot_script
│       ├── filesystem
│       │   ├── boot
│       │   │   └── README
│       │   ├── home
│       │   │   ├── pi
│       │   │   │   └── README
│       │   │   └── root
│       │   │       └── README
│       │   └── root
│       │       └── README
│       └── start_chroot_script
└── vagrant
    ├── run_vagrant_build.sh
    ├── setup.sh
    └── Vagrantfile

I'll do my best to explain what these files allow us to do, but I won't be able to touch on everything in detail.

Let's start with...

The Config File

The config file contains distro-wide settings. For those creating an Ubuntu Server spin-off, it should look something like this.

export DIST_NAME=CustomDisto
export DIST_VERSION=0.0.1
export MODULES="base(network,docker,custom_distro)"


export BASE_DISTRO=ubuntu
export BASE_ARCH=arm64

export BASE_ADD_USER=yes
export BASE_USER=pi
export BASE_USER_PASSWORD=12345

export BASE_IMAGE_ENLARGEROOT=4000
export BASE_IMAGE_RESIZEROOT=200

Some of these variables are self-explanatory, but some are not. The MODULES variable defines how functionality will be added to the base image (we'll talk more about that in a moment). The BASE_IMAGE_ENLARGEROOT and BASE_IMAGE_RESIZEROOT variables determine how the size of the image is increased before and after files are copied onto it (these two variables are optional; only use them if you need to).

The Image Folder

The image folder is where the base image for our custom distro should go. In fact, if you're following along, go ahead and download the 64-bit Ubuntu Server disk image file. It can be found here. Don't extract the image, though. Simply place the compressed file in the image directory.

CustomPiOS should be able to find most images on its own, but it has trouble with the 64-bit Ubuntu image. If it's having trouble, set the base image path in the config file like so.

export BASE_ZIP_IMAGE=image/my_base_image.img.xz

The Modules Folder

The modules folder contains "modules" that add functionality to the custom distro. By default, a module of the same name as your custom distro folder was created when you ran the make_custom_pi_os script. Within this module is a filesystem folder, a start_chroot_script, an end_chroot_script, and another config file.

Let's break each of these down.

The Filesystem Folder

The filesystem folder contains all of the custom files that you would like to be present on your custom distro. It contains three sub-folders, each of which will unpacked at a different mount point: the root folder will unpacked at the root directory, or /; the home folder will be unpacked at /home; and the boot directory will be unpacked in the boot partition (which is usually mounted at /boot).

The Start Chroot Script

The start_chroot_script runs in a chroot environment on the disk image. Use this to install software, change file permissions, and/or do anything else that would require the use of terminal.

In this script, you should also have access to a few extra commands (courtesy of the auto-generated line source /common.sh): the unpack command and the gitclone command. More information can be found here.

The End Chroot Script

The end_chroot_script can be used to clean up after the start_chroot_script. A lot of modules don't use this, however. Feel free to delete it if you'd like.

The Module Config File

The config file within the module folder can used to set environment variables for your own use within the start and end chroot scripts. By default, only an example variable is generated.

CUSTOM_DISTRO_VAR="This is a module variable"

Note that the variable name starts with CUSTOM_DISTRO. For a variable to be exported, it's name must begin with the module name. Also, note the slight change in syntax from the global config file. Instead of using the export command, these config files use .env style syntax.

Although you could hard-code values into the start and end chroot scripts, I would strongly recommend using these module-level config files. The variables exported from these files are also accessible in (and can be overridden by) the main config file. This greatly improves modularity; after you're done creating your custom distro, you could even release your custom_distro module as an add-on to CustomPiOS.

Including Modules in the Final Disk Image

Now, let's revisit this line in the main config file.

export MODULES="base(network,docker,custom_distro)"

The variable MODULES defines which modules to include and how those modules should depend on each other. In the example above, three modules are included: base, network, docker, and custom_distro. The network, docker, and custom_distro modules depend on (or extend, we could say) the base module. In fact, every module must depend on the base module. However, it can get more complicated. Here's another example.

export MODULES="base(network(octopi, picamera))"

In this case, octopi and picamera depend on network, which depends on base.

I would also like to point out that we have access to quite a few built-in modules, including...

admin-toolkit
auto-hotspot
auto-mount-removable
cockpit
docker
gui
usbconsole

Those are a few of my favorites. A full list with descriptions can be found here.

Building our Custom Ubuntu Server

In order to create our custom Ubuntu Server, we're just going to make a few changes to the custom_distro module.

Setting a Static IP

In order to set a static IP, we'll create etc/netplan/00-installer-config.yaml within the filesystem/root directory of our module. Within this file, we'll save the following,

# Static ip configuration
network:
  ethernets:
    eth0:
      addresses: [10.0.0.2/24]
      gateway4: [10.0.0.1]
      nameservers:
        addresses: [4.2.2.2, 8.8.8.8]
  version: 2

where 10.0.0.2 is our static ip.

Configuring docker-compose

To configure docker-compose, simply create a folder called docker-compose within the root directory, and then place your docker-compose.yml file within that folder along with another configuration files you might need (like a .env file).

First-time Boot Script

It's often helpful to have docker-compose start on boot. A restart flag can be set in the docker-compose.yml file, but docker-compose still won't start on the first boot-up. To do that, we'll write first-time boot script.

In the filesystem/root/etc directory, create a file named rc.local. Then, within that file, save the following contents.

#!/bin/bash

echo 
echo "==== Starting Docker Compose ===="
echo

pushd /docker-compose
  docker-compose up -d
popd

echo
echo "==== Docker Compose has been started ===="
echo

rm ${BASH_SOURCE[0]} # Remove this line if you want the script to run on every boot-up

Finally, we'll need to make this script executable. We can do that easily within the start_chroot_script by adding the following line to the end of the script.

chmod +x /etc/rc.local

Building the Image

After finishing the configuration of our custom distro, we finally get to build our distributable image. This can be done by running the following command (make sure you're in the custom_distro/src directory).

sudo ./build_dist

After the command finishes (it can take some time), the resulting distributable image can be found in the newly created custom_distro/src/workspace directory. It'll have the same name as the base image.

If you encounter any errors, be sure to review the configuration files. You may also want to try one of the different build methods listed on the CustomPiOS github page.

Conclusion

CustomPiOS is a really powerful tool, and I'm actually surprised there aren't more tutorials for it out there. It's been really useful to me, and I think a lot of other people would appreciate knowing how to use it.

To check out some of the other projects that use CustomPiOS, take a look at this list.

Thanks so much for taking the time to read this article. I know it's kind of a long one, but I hope it's helpful. If you have any questions or if you find any errors in this article, please let me know in the comments below.

Have a good one,
- Simon

SDG

.
.
.
.
.
Photo by Harrison Broadbent on Unsplash

Iterating through trees using the yield statement

Simon Richard — Sat, 12 Dec 2020 17:03:35 +0000

Data is often organized using tree like structures; JSON, XML, and File Directories are good examples of this. In this post, I would like to talk about how the keyword yield (in combination with recursion) can be used to iterate easily through complex¹ tree like structures.

¹ By complex, I mean multi-typed and messy.

The `yield` Statement

For those who are not familiar with yield statements in python, here's a quick explanation.

The yield statement allows a function to return a value and then continue running. In practice, this means turning the function into an iterator. For example:

def foo():
    n = 0
    while True:
        yield n
        n += 1

my_iter = foo()
my_iter.next()
> 0
my_iter.next()
> 1
my_iten.next()
> 2
...

We can also use a for loop.

my_iter = foo()
for n in my_iter:
    print(n)
> 0
> 1
> 2
...

Recursion

Recursion occurs when a function is executed inside of itself. This idea is especially helpful in relation to self similar structures, like trees. However, that function needs an exit point in order to actually be useful. Otherwise, it would run forever and return nothing. Let's look at some examples.

This function will run forever.

def foo(x):
    return 1 + foo(x)

This function will not run forever. Notice that it has an exit point, which is used when x >= 5.

def bar(x):
    if x < 5:
        return var(x+1)
    else:
        return x

This function prints all of the items of a nested array.

def print_nested_array(x):
    if isinstance(x, list):
        for item in x:
            print_nested_array(item)
    else:
        print(x)

That one's a little more complicated; let's talk about it.

The first value that's passed through this function is probably going to be an array (since we called the function print_nested_array). In that case, the function will loop through that array and pass each of its items back into the function.

For each item that is not an array, the function will print its string representation to the console. That's our exit point.

If one of those items is also an array, it will go through the same process the initial array went through. Each of its items will be passed back into the function.

Let's test it.

arr = [ [1,2], 3, [[4], 5, 6], 7]
print_nested_array(arr)

Using Recursion and `yield`

The yield statement allows us to pass each of the items in our nested array back to the first instance of our function (the one that was not called using recursion), making it easier for us to access those values later on.

To implement this, we'll only need to do few things to the previous function we used (shown below for convenience).

def print_nested_array(x):
    if isinstance(x, list):
        for item in x:
            print_nested_array(item)
    else:
        print(x)

First, we'll replace the print statement with a yield statement.

Then, since our function now returns an iterator, we'll also have to handle that inside our for loop. In this case, instead of just calling our function again, we'll use another for loop to iterate through the iterator it returns. Each of the items in that loop will also be yielded.

def parse_array(x):
    if isinstance(x, list):
        for item in x:
            for result in parse_array(item):
                yield result
    else:
        yield x

Let's test it.

arr = [
    "foo",
    "bar",
    ["asdf", 34],
    [[], [1, 2]],
    [3, 4, 5, 6, 7],
    True
]

for item in parse_array(arr):
    print(item)

foo
bar
asdf
34
1
2
3
4
5
6
7
True

We could have done that with our previous function. However, we can also cast the iterator the function returns into a list.

list( parse_array(arr) )

["foo", "bar", "asdf", 34, 1, 2, 3, 4, 5, 6, 7, True]

Let's modify the function so we can print a nicely formatted representation of the array.

def print_array(x):
    if isinstance(x, list):
        for item in x:
            for result in print_array(item):
                yield "  "+str(result)
    else:
        yield str(x)

for row in print_array(arr):
    print(row)

What if our array also had dictionaries in it? What if it was a dictionary? We can add an elif statement to handle that.

def parse_something(x):
    if isinstance(x, list):
        for item in x:
            for result in parse_something(item):
                yield result
    elif isinstance(x, dict):
        for key in x:
            for result in parse_something(x[key]):
                yield result
    else:
        yield x

To print a nicely formatted version of our array/dictionary, we could use this function.

def print_something(x):
    if isinstance(x, list):
        for item in x:
            for result in print_something(item):
                yield "  "+str(result)
    elif isinstance(x, dict):
        for key in x:
            yield ""
            yield key+":"
            for result in print_something(x[key]):
                yield "  "+str(result)
    else:
        yield str(x)

Let's test it.

my_dict = {
    "key1": 1,
    "key2": [1, 2, {"foo": 42}],
    "bar": {
        "a": 1234,
        "b": 5678
    }
}

print_something(my_dict)


key1:
  1

key2:
  1
  2

  foo:
    42

bar:

  a:
    1234

  b:
    5678

Conclusion

The combination of recursion and the yield statement is pretty powerful. I'm definitely going to keep it in the back of my head when I'm programming.

I hope you've also found this post to be helpful (or at least a little interesting).

— jsimonrichard

Repl IT Post

DEV Community: Simon Richard

MongoDB Hackathon Submission - QrTracker

Overview of My Submission

Submission Category:

Link to Code

jsimonrichard / QrTracker-atlas-hackathon

Submission for the Atlas Hackathon put on by MongoDB and dev.to

Screenshots

MongoDB Technologies Used

Other Technologies Used

Creating a Custom Ubuntu Server Disk Image for the Raspberry Pi with CustomPiOS

Getting Started

The Config File

The Image Folder

The Modules Folder

The Filesystem Folder

The Start Chroot Script

The End Chroot Script

The Module Config File

Including Modules in the Final Disk Image

Building our Custom Ubuntu Server

Setting a Static IP

Configuring docker-compose

First-time Boot Script

Building the Image

Conclusion

Iterating through trees using the yield statement

The yield Statement

Recursion

Using Recursion and yield

Conclusion

The `yield` Statement

Using Recursion and `yield`