DEV Community: Jonathan Boudreau

Differences Between MacOS and Linux Scripting

Jonathan Boudreau — Thu, 22 Oct 2020 00:00:33 +0000

Photo by Sharon McCutcheon on Unsplash

I often hear people saying MacOS and Linux are similar. I would like to propose that they are in fact very different.

But why does this matter? Well, for one, I write a lot of code for CI (Continuous Integration). The differences are big enough to make it challenging to re-use more complex scripts locally and in CI.

The Shell

MacOS recently changed its default shell to zsh. Before this, it was using a decade old release of bash.

Now, nothing is stopping you from installing the latest version of bash. If you're writing scripts for your team it is unlikely you will have much control over that however.

Package Manager

This is probably the most significant difference with MacOS and Linux when it comes to scripting automation. Most people on Mac rely on Homebrew, while most servers on Linux use apt. Not only is it different to install packages, but so is adding new software sources.

Docker

Since Docker for Mac runs in a virtual machine (using linuxkit), there are significant differences with running containers on it.

Volumes (bind mounts) have poorer performance. This may limit how many things you can throw at compose if your project is a large codebase.

File permissions map differently from the container to the host. Linux may use subordinate uids and gids, whilst MacOS maps all users to one uid/gid on the host.

find

MacOS uses a different implementation of find. Linux has a couple of extra options, and the search path is optional.

netstat

Similarly to find, netstat on Linux is a different implementation altogether. For certain tasks you will need to use lsof when on MacOS. The output for lsof is also different, so you will need to implement different parsing code for each operating system.

ifconfig

ifconfig is deprecated on Linux. It is recommended to write new scripts with ip instead. Since MacOS doesn't have ip, you're stuck with writing an implementation for each operating system.

xargs

xargs, which is used for piping arguments to another command, works slightly differently on MacOS. If the previous command has no output, xargs on MacOS won't run the command.

init

MacOS uses launchd, which accepts an xml configuration for the definition of new services. Most modern Linux distributions rely on systemd, which has a INI-like configuration format.

In Closing

Did you run into other discrepancies between the two operating systems? What are some of the best practices you follow to write portable scripts?

Why is Linux Not More Popular on the Desktop?

Jonathan Boudreau — Sun, 13 Jan 2019 17:06:24 +0000

Picture taken from wikipedia.

There are issues when it comes to Linux being more broadly adopted on the desktop. What are the main issues you think and how can they be addressed?

I think that the big issue with Linux is fragmentation. This has been especially an issue when it comes to developers publishing games on Linux.

The solution to this issue are cross-distribution installers like Snap, AppImage and Flatpak. Every application has its own set of libraries, meaning they do not depend on distribution-specific packages.

Another issue I believe is momentum. There's some outdated opinion on how well Linux works on the desktop.

Defeating the momentum will require more effort on the PR front. I don't see this being something that will just fix itself with time.

Thoughts?

My Ubuntu Desktop

Jonathan Boudreau — Fri, 28 Dec 2018 16:00:58 +0000

I recently reinstalled Ubuntu on my personal laptop. I found some pleasant surprises in doing so, and I wanted to share what I've got going.

Note: This is my current setup for Ubuntu 18.10. Your mileage may vary if on LTS.

Hotkeys

The default hotkeys in GNOME are pretty good out of the box. Here are some that I use on a regular basis:

alt + tab: Switch of program.
alt + backtick: Switch to a window which belongs to the same program.
ctrl + alt + arrow: Move to a different workspace. Workspaces in GNOME can only go up or down.
ctrl + alt + shift + arrow: Move current window to a different workspace.
ctrl + alt + t: open gnome terminal
meta (windows button): Open the global menu, I use it to search for programs instead of relying on the mouse.
meta + left or right: move the window left or right.
meta + up: make the window take up the entire viewport.
shift + printscr: Select a region to do a screen capture.
alt + printscr: Do a screen capture of the entire focused window.

There's really just two things that bother me with the default hotkeys. First, program switching is for all programs, across workspaces. This isn't an issue if you don't normally have a lot of programs open, but I tend to multitask and prefer keeping things isolated between workspaces. Second, I like to switch of workspaces using using indexes (e.g., super+2 to switch to workspace number 2). This is something I got used to from back when had i3wm as part of my main desktop environment.

Limiting application switching to workspaces

To make this customization we will need the DConf Editor. It can be installed through the Ubuntu Software program. Simply open up "Ubuntu Software", search for "Dconf" and install the "Dconf Editor":

After opening the DConf Editor, navigate to the path org -> gnome -> shell -> app-switcher -> current-workspace-only. Disable the "Use default value" option and set "Custom value" to "true".

Changing of workspace using indexes

This step is easier since its just a regular setting. Open up the "Settings" application, select "Devices", and click the "Keyboard" option. Change the "Switch to workspace" option to the desired keybinding.

Look and Feel

In Ubuntu 18.10, the default theme is actually quite good out of the box. I personally didn't change it this time around, but for those looking for alternatives there are quite a few solid options out there such as arc and adapta themes.

I have only a few issues with the out of the box look and feel:

Dock is in the way (if you have a keyboard-driven workflow)
Some of the animations are too slow
Terminal menu bar isn't needed (you can get the menu by right clicking)

Removing the dock

Unfortunately, you can't disable the dock from the Tweaks program. Instead, we need to uninstall it entirely. In the terminal type the following:

sudo apt remove gnome-shell-extension-ubuntu-dock

You will need to re-log into your machine for the changes to take effect.

If you want to bring back the dock, simply run:

sudo apt install gnome-shell-extension-ubuntu-dock

Speeding up animations

To reduce the time of animations, all we need is to grab an extension from the Ubuntu Software program. Simply search for "impatience" and it should pop up.

Install the extension and open the settings for it. I've personally reduced the animation time in half.

Disabling the terminal menu bar

Open the gnome terminal, right click, select preferences. Go to the general section and you should untick the "Show menubar by default in new terminals" checkbox.

Additional Software

When installing additional software, it is preferable to do it through software repositories instead of relying on archives (.deb files). Installing programs through repositories means the operating system will handle upgrading programs for you.

All of the additional software I need can be simply installed through the "Ubuntu Software" program:

Spotify
Discord
Steam

Since this is for my personal laptop, there isn't much in terms of office software that I setup. Most of it is browser-based in my case.

Development

I've already written an article on how I develop from containers. However, I still need to install custom fonts and configure the terminal profile.

To install powerline patched fonts:

git clone https://github.com/powerline/fonts /tmp/fonts
cd /tmp/fonts
./install.sh

Go to the terminal preferences and select the correct profile in use. If you haven't changed the terminal profile this will be the "Unnamed" profile. Select the "Custom font" checkbox and click on the textbox beside it to pick a different font. The font should have "for Powerline" in it, otherwise some glyphs will be missing.

Conclusion

Setting up a new machine is pretty simple and surprisingly doesn't require me to use the terminal all that much. While I may be extremely comfortable in the terminal (caugh, NeoVIM user ;)) it is encouraging to see the progress that has been made towards a more approachable desktop. Hopefully one day I'll be able to use Linux at my work.

Why Hash Maps are Randomized

Jonathan Boudreau — Fri, 30 Nov 2018 02:09:25 +0000

I implemented a hash map (hash table) on one of my side projects recently just for fun and learned an interesting tidbit about security that I wasn't aware of. Figured I'd share it!

How Hash Maps Work

At its core, the idea behind a hash map is that you convert a given key (e.g., a string) into an index value using a hash function. The index will then determine the location of your data on an array. Insert, delete, and read have a constant time complexity (best case 😉). This means that no matter how many elements you have in your hash map the time it takes for your lookups shouldn't really change.

However, since we are compressing information into a single integer, it is possible to have collisions. The solution is to use another array (a bucket), inside of our hash map's array, to hold all collisions.

I'll be skipping over some details, but just to get an idea this is what a hash map implementation would roughly look like:

class HashMap
    constructor() {
        this.buckets = new Array(1000) // arbitrary
    }
    set(key, value) {
        const index = hash(key)
        const bucket = this.buckets[index]
        if(!bucket) {
            this.buckets[index] = [{
                key: key,
                value: value
            }]
        } else {
            const item = bucket.find(item => item.key === key)
            if(item) {
                item.value = value
            } else {
                bucket.push({
                    key: key,
                    value: value
                })
            }
        }
    }
}

Denial of Service

Denial of service broadly refers to an attack where the service is brought down either temporarily or permanently.

Quite often the way this is done is by triggering a highly expensive operation on the server side repeatedly.

The Attack

As was mentioned above, when collisions occur more items will be placed into the same bucket. If it happens enough times the performance of your hash map will start to look more like a sequential array of keys and values. In other words the complexity will become linear.

Depending on how critical the performance of your hash map is, your server resources may be taken up easily.

The Solution

For the attacker to cause collisions in our hash map, he needs to know ahead of time what will cause them. Our solution revolves around making it nearly impossible for the attacker to guess the correct combination of keys to cause enough collisions which would impact the performance of the server.

Some hash functions allow a seed to be specified for the hash. When the hash map is created we generate the random seed. This means that each hash map instance will convert the same key to a different index. If it is a cryptographically secure random number generator, we know that the attacker will not be able to guess it.

Here is the rough idea for the solution:

class HashMap
    constructor() {
        this.seed = random()
        this.buckets = new Array(1000) // arbitrary
    }
    set(key, value) {
        const index = hash(this.seed, key)
        const bucket = this.buckets[index]
        if(!bucket) {
            this.buckets[index] = [{
                key: key,
                value: value
            }]
        } else {
            const item = bucket.find(item => item.key === key)
            if(item) {
                item.value = value
            } else {
                bucket.push({
                    key: key,
                    value: value
                })
            }
        }
    }
}

Developing from Containers

Jonathan Boudreau — Thu, 29 Nov 2018 12:50:48 +0000

Using containers for development has become a widespread practice. The common use case in development is for running services required by the application. Things such as installing Redis, MongoDB, or even Elasticsearch. Most of the time, developers rely on docker-compose to define the whole set of services required for the application.

I've taken things further by creating container images with everything I need to develop in a given language. This includes (but is not limited to):

Code editor
Debugger
Profiler
Tracing tools
Package manager
Build tools

Motivation

OS Independent

Even if you're using Docker for Mac, you still have a Linux-based environment to develop from. Its really good for getting familiar with Linux, which can then be applied to creating real production container images (among many other things).

Installation Speed

Compared to using installer scripts, container images (in particular docker images) are really quick to get going on a new machine. This is because you only need to download the image with pre-compiled binaries. For example, I've had to compile the cquery server from source for one of my images. If I was using an installer script I'd have to either host the compiled binary myself or compile it on each machine which takes quite some time (since you need to download clang and all that jazz).

Portability

If using an installer script, things might not work on a different machine because certain packages were installed in a different manner. For example, someone might've used a PPA to install it one machine, while on another machine it was installed as a .deb archive or using the standard software sources. In contrast, once a container image is built it will just work on practically any machine.

Reproducibility

Once the image is built, it won't break out of nowhere. This is unlike installer scripts; I've had many occurrences back when I was just using a git repository to store configurations and scripts where it would break based on when the installer would be run.

Caveats

Not a Common Use Case for Docker

Support from docker for this sort of workflow hasn't been stellar. Some issues with docker have taken years to fix.

Still Needs Some Glue Code

There are several things which you want to map from the host into the container. I've written a tool to make this easier called slipway. Before I wrote this tool I would use a shell script to initialize the container.

If you're not using Linux as the host operating system, things can get even more complicated.

Anatomy of a Containerized Development Environment

0. Pick a base image

For the uninitiated, building a docker image requires a configuration file (a Dockerfile) which defines steps to be executed in a container. After all of these steps are run, the container is "committed" as a new image.

Most images are based off another image. I recommend starting out with an Ubuntu base since it is very popular.

Add a file named Dockerfile in an empty directory:

FROM ubuntu:bionic

1. Set Up Your User

The image only comes with a root user. We'll need to create a user with appropriate permissions to run inside of the container.

# Feel free to change this to whatever your want
ENV DOCKER_USER developer

# Create user with passwordless sudo. This is only acceptable as it is a
# private development environment not exposed to the outside world. Do 
# NOT do this on your host machine or otherwise.
RUN apt-get update && \
    apt-get install -y sudo && \
    adduser --disabled-password --gecos '' "$DOCKER_USER" && \
    adduser "$DOCKER_USER" sudo && \
    echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers && \
    touch /home/$DOCKER_USER/.sudo_as_admin_successful && \
    rm -rf /var/lib/apt/lists/*

USER "$DOCKER_USER"

WORKDIR "/home/$DOCKER_USER"

2. Basic Tools

Ubuntu core (which is what our image is based on) does not include much since it is built for "ready to ship" applications. Lets add some basic packages for development:

RUN yes | sudo unminimize && \
    sudo apt-get install -y man-db bash-completion build-essential curl openssh-client && \
    sudo rm -rf /var/lib/apt/lists/*

3. Multiplexing (Optional)

If you aren't using a GUI editor like VSCode or Webstorm, you'll probably want a program which can take a single shell session and split it into multiple ones. This is called a terminal multiplexer. I prefer tmux.

RUN sudo apt-get update && \
    sudo apt-get install -y tmux && \
    sudo rm -rf /var/lib/apt/lists/*

There's plenty of customization options which I could discuss here, but to keep the tutorial short I will be skipping over this step. Feel free to check out my project for riced up development environment ideas.

4. Install Your Editor of Choice

Now we need something to edit source code. I can recommend Neovim, but any editor will do.

RUN sudo apt-get update && \
    sudo apt-get install -y neovim && \
    sudo rm -rf var/lib/apt/lists/*

As above, I am skipping customization options.

5. Install Your Runtime

In this example, we'll be installing NodeJs since most developers on the site use it. If you're using Python or some other language at this step you can install it instead.

RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash
ENV NVM_DIR /home/$DOCKER_USER/.nvm

RUN . "$NVM_DIR/nvm.sh" && \
    nvm install --lts && \
    nvm alias default stable

Install any additional tools for your language of choice at this step.

Building, then running

All you need is to build the image before you can run it.

In your terminal, run:

docker build -t development-environment .

The final command to run your environment can vary based on what you want to transfer over. Here is an example command:

docker run --rm -ti \
    -v $HOME/workspace:/home/developer/workspace \
    development-environment bash

For a more detailed version of this tutorial, see my repository's tutorial section.

Hi, I'm Jonathan Boudreau

Jonathan Boudreau — Mon, 02 Jan 2017 05:51:59 +0000

I have been coding for 3 years.

You can find me on GitHub as AGhost-7

I mostly program in these languages: Javascript, Scala, Rust.

I am currently learning more about DevOps.

Nice to meet you.