DEV Community: Bjarne Magnussen

A Primer To Go Modules

Bjarne Magnussen — Wed, 01 Apr 2020 13:00:48 +0000

(Image from https://www.youtube.com/watch?v=NCAIhepJgr0)

Go Modules was introduced in 2018 with Go version 1.11 and with the current (25th of February 2020) version 1.14 is considered ready for production use. It is fairly straight-forward and easy to use. This blog post gives some background on the historical transition to modules, what modules are and the advantages of using it.

What are Go Packages?

Modules use the existing package management of Go without changing it fundamentally. To better understand them it is therefore necessary to first understand what packages are.

An important and fundamental part of high-quality software is code reuse embodied in the principle “Don't Repeat Yourself” — the DRY principle. You should never repeat the same code over and over again, but instead allow it to be reused as much as possible. This also increases the maintainability of code.

Packages help in organizing related Go source files together into an atomic unit. A package bundles functions and variables together and makes them available to the rest of a bigger project or to completely separate projects. This makes it easier to share code with other projects by allowing them to import and reuse it.

In this way Go encourages you to write small pieces of software components through packages, and compose your applications with these small packages.

Using packages has the following advantages:

No name conflicts, since functions can have the same name while belonging to different packages.
Allows reusing and sharing code among projects.
Organizes related code together.
Reduces compile times as packages are only re-compiled if their code is changed.

Declaring a package

A package is not much more than a directory inside your Go workspace with one or more Go source files, or other Go packages.

(Image from https://www.callicoder.com/golang-packages/)

Every Go source file must belong to a package, which is declared at the top of a file using the following syntax:

package <packagename>

All functions and variables defined in this source file then become part of the declared package.

You can export a member of your package by letting its declared name begin with a capital letter. Only then can other packages import and use it.

Importing packages

To import packages in Go use the import syntax

import (
  "fmt"
  "time"
  "math"
  "math/rand"
)

The last element of the import path is the package name available for your project. For example, the package imported with math/rand is simply rand and its exported function Intn(n int) can be invoked with e.g. rand.Intn(100). The package is imported with math/rand because it is nested inside the math package as a subdirectory.

Nesting

It is possible to nest a package inside another by creating a subdirectory.

$GOPATH
└── github.com
    └── bjarnemagnussen
        └── myproject
            ├── numbers          # Some Package
            └── strings          # Another Package
                └── greet        # Nested Package
                    └── texts.go

The GOPATH

All import paths for non-builtin packages are relative to its GOPATH.

import (
  "github.com/bjarnemagnussen/myproject/numbers"
  "github.com/bjarnemagnussen/myproject/strings"  
  "github.com/bjarnemagnussen/myproject/strings/greet"
)

Getting 3rd Party Packages

This makes it very easy to import third party packages. You can use the go get command to download third party packages from remote repositories.

$ go get -u github.com/jinzhu/gorm

The above command fetches the gorm package from Github and automatically stores it inside GOPATH. To use it import the package to any of the source files in your project with:

import "github.com/jinzhu/gorm"

Package Alias

What happens if different packages have the same names? Package alias can resolve naming conflicts between packages, or give a short name to an imported package.

import (
  str "strings"   // Package Alias "str"
)

Vendoring And Versioning

Go packages do not provide any builtin versioning system. The Go FAQ suggested to create a local copy of a third party package that should stay frozen:

If you're using an externally supplied package and worry that it might change in unexpected ways, the simplest solution is to copy it to your local repository. (This is the approach Google takes internally.) Store the copy under a new import path that identifies it as a local copy.

Over the years many different tools where proposed and developed around the idea of “vendoring” such as vend, gb and dep. Around 2016 a discussion arose to tag versions onto packages and prepare for a time when it would become valuable. The community agreed on git release tags with semantic versioning.

Semantic versioning divides releases into

a major (breaking) version,
a minor (feature) version,
and a patch version.

The full version is prepended with a single “v” and looks like “v1.2.3”, which translates into a major version 1, with minor version 2 and patch version 3. When there is a bugfix the patch version is increased. If features are added that are backwards compatible the minor version is increased. For changes to the code that remove backwards compatibility the major version is increased.

Version tags are used by vendoring tools such as dep to define which commit state of a package to vendor. However, another approach was introduced in 2018 to completely eliminate vendoring and allow for project-based workflows instead of GOPATH: Go Modules.

Go Modules

Today Go modules has replaced dep and followed up on the work of yet another dependency manager called vgo. Modules is a way of packaging software and is Go's new simplistic dependency management that even allows for reproducible builds.

Imagine you are developing a Go program with a number of dependencies such as package A. At the time of writing your program, package A works in a set way.

However, what happens when the maintainers of package A update their program to fix a bug or extend functionality? You might get lucky and their changes might not impact your application. However, you might be unlucky and these changes break your application.

This is where Go Modules comes in to save the day. By using modules, we can select the precise versions of a dependency that we wish to use and ensure that whenever we build our program, it always uses the specified versions.

go.mod

A module is just a collection of Go packages stored in a directory with a go.mod file at its root. As an example the following go.mod file defines the module github.com/my/thing:

module github.com/my/thing

require (
    github.com/some/dependency v1.2.3
    github.com/another/dependency/v4 v4.0.0
)

There are four directives: module, require, replace and exclude. I refer to the excellent Go Wiki page to read about the last two directives replace and exclude. The require directive specifies the versions of the named packages that are used by this module as dependencies and will be revisited in more detail later.

A module path is defined with the module directive. All packages in a module share the module path as a common prefix to their import paths. In other words, the module path and relative path from the go.mod file to a package's directory decides its import path.

For example, if you are creating a module for a repository github.com/my/thing with two packages foo and bar, then the first line of the go.mod file would declare the module path with module github.com/my/thing. The corresponding directory structure could be:

thing
├── bar
|   └── bar.go
├── foo
|   └── foo.go
└── go.mod

The import paths for the foo and bar packages would then be github.com/my/thing/foo and github.com/my/thing/bar. Hence the package foo would be imported in a Go source file with:

import "github.com/my/thing/foo"

Importing a module stored locally outside your GOPATH from another module can be easily achieved using a VCS (e.g. github.com) for the dependency.

Go can automatically take care of downloading and importing a dependency if it is using a VCS, regardless of where its working directory is stored on disk. Alternatively, the replace directive of go.mod can be used to point to the dependency's local directory, see this answer on StackOverflow.

Version Selection

With Go Modules, adding a new dependency to any of your Go source files will have most Go commands (such as go build and go test) automatically download the highest version of that new package and add it as a direct dependency to the go.mod file with the require directive.

For example, assume you create a new module from scratch with a go.mod file only consisting of:

module github.com/my/thing

Adding a new import to a dependency github.com/some/mod, which latest tagged release version is v1.2.3, and running go build ./... will automatically download the dependency and change your go.mod file to reflect the new dependency with its latest version:

module github.com/my/thing

require (
  github.com/some/mod v1.2.3
)

The module github.com/some/mod is now a dependency with an allowed version greater than v1.2.3 but less than v2, since the assumed semantic version means that v2 is incompatible with v1.

Go Modules uses a minimal version selection for indirect dependencies. That means that Go Modules selects the highest of the versions explicitly listed by a require directive in your module or any one of its dependencies. If your module depends on A that itself has a require D v1.0.0 and your module also depends on B that has a require D v1.1.1, then Go Modules would select v1.1.1 of dependency D. This selection of v1.1.1 remains consistent even if some time later a v1.2.0 of D becomes available. This is in contrast to go dep, which always selects the highest compatible released version under the assumption of semantic versioning.

To see a list of the selected module versions (including indirect dependencies), you can use go list -m all.

If later an indirect dependency is removed, Go Modules will still keep track of the “latest not-greatest” version. In other words, if we were to remove from our module the dependency B containing a require D v1.1.1 but keep dependency A with a require D v1.0.0, then Go Modules would not fallback to v1.0.0 but instead keep v1.1.1 of D.

Keeping track of dependency versions is done with a go.sum file.

go.sum

The go.sum file is an opaque reliability meta data file and it should not be used to understand your dependencies. Any time your Go Module downloads a version of a dependency not yet seen, that version is tagged inside the go.sum file with its corresponding git commit hash acting as a cryptographic checksum.

The go.sum file retains checksums for dependency versions even after you stop using that particular dependency or version. This allows validation of the checksums if you later resume using something, providing additional safety. The go.sum will therefore frequently have more dependency listed than your go.mod file.

Your module's go.sum file should always be committed along with your go.mod file.

Committing go.sum has the following benefits:

If someone clones your repository and downloads your dependencies, they will receive an error if there is any mismatch between their downloaded copies of your dependencies and the corresponding entries in your go.sum.
In addition, go mod verify checks that the on-disk cached copies of module downloads still match the entries in go.sum.

Activating And Using Modules

Enough of the theory! Let's see how we can add Go Modules to a project and use it.

(Image from https://blog.francium.tech/go-modules-go-project-set-up-without-gopath-1ae601a4e868)

One of the main features of modules is to work on Go projects outside of GOPATH. We will therefore start by navigating to the root directory of some imaginary project not stored in the GOPATH

$ cd <project path outside $GOPATH/src>         # e.g., cd ~/projects/hello

Create the initial module definition by automatically writing it to the go.mod file with:

$ go mod init

go mod init will often be able to use auxiliary data (such as VCS meta-data) to automatically determine the appropriate module path. But if go mod init should state that it cannot automatically determine the module path, you can supply the module path as an optional argument with go mod init github.com/my/thing.

Build the module with

$ go build ./...

Building the module will automatically download all necessary dependencies and choose their latest versions. The direct dependencies will show up with the require directive inside go.mod, while both the direct and indirect dependencies are tagged to their commit hashes inside go.sum.

It is good practice to run go mod tidy before committing code to your repository.

This will ensure that the go.mod file contains the dependency requirements for all possible combinations of OS, architecture, and build tags, including for testing purposes. This will help others on your team and your CI/CD environments.

Running go mod tidy will also fix any inconsistency reflected by a missing go.mod file of a dependency of your module, e.g. because the dependency has not yet opted in to modules itself, or if its go.mod file is missing one or more of its dependencies, e.g. because the module author did not run go mod tidy. The missing transitive dependencies will be added to your module's requirements along with an // indirect comment to indicate that the dependency is not from a direct import within your module.

This behaviour is how modules provide 100% reproducible builds and tests by recording precise dependency information.

Updating Dependencies

To update a specific dependency inside your module and all its indirect dependency you run as usually

$ go get -u github/some/package

This will also automatically change both go.mod and go.sum to reflect the new version(s).

To upgrade all direct and indirect dependencies to their latest versions do the following:

Upgrading to latest minor or patch releases: go get -u ./... (and add -t to also upgrade test dependencies).
Upgrading to latest patch releases only: go get -u=patch ./... (and add -t to also upgrade test dependencies).

Conclusion

I may have skipped a lot of small details in this post. But the idea behind Go Modules is that it should often be possible to just magically work without configuration. However, knowing some background from where Go Modules comes helps in understanding the design choices. Today Go Modules should be used with every new project and can also easily be activated for existing ones.

Enum vs Flag for bitmasks in Python

Bjarne Magnussen — Thu, 28 Nov 2019 17:16:21 +0000

Ever wondered what's behind those funny looking Unix access permission values like 600 ✋, 777 ⚠️, or 666 😈? Well, they are octal representations, where each of the three rightmost digits represents a different part of the permissions for owner, group, and others.

Each of these digits is created from the sum of its component bits. As a result, specific bits add to the sum as it is represented by a numeral:

The read bit adds 4 to its total (in binary 100),
The write bit adds 2 to its total (in binary 010), and
The execute bit adds 1 to its total (in binary 001).

These values never produce ambiguous combinations and therefore each sum represents a unique set of permissions. Let's take 600 as an example. The first digit 6 represents the permissions for the owner. In binary it is 110 and the only way to create it is from the sum of component bits above by adding 010 (write access) to 100 (read access). In other words, the owner has permission to read and write the file, but not to execute it.

Using this system for permissions is optimized for data storage. But also turning permissions on (or off) can be done efficiently in what is known as bit-masking.

Bitmasks

You see, to turn certain bits on, the OR operator can be used. This operator follows the rule that X OR 1 = 1 and X OR 0 = X. If from the example above we now also want the owner to have executive permission on the file, we simply use OR with 1 (execute value) on the current permission value: 6 OR 1 = 7, or in binary:

We can also turn off certain bits by using the AND operator, which follows the rule that X AND 1 = X and X AND 0 = 0. To turn off a bit we use it with 0. Interestingly we can leave the bit as it was when using 1.

Turning bits on and off is also referred to as "masking on" and "masking off".

Using this very old technique to store permission parts is very cool! But are there other use-cases where it makes sense? There are still today a ton of them, although they have become unpopular due to much more efficient hardware allowing the use of a set or dictionary to store the representations that a bit would otherwise hold.

Filtering using bit masking

For the Bit project I needed a way to store the supported features for each of the numerous objects representing network APIs. Instead of using a dictionary to map each API to its supported set of features, or store a set of features on each API, I chose to use a system similar to the Unix access permission values.

Making use of bit masking allows for fast filtering of APIs depending on specific features. This can be achieved by using a single AND operator. We just use a filter value Y that represents the features we want to filter for, and the value X representing the supported features of some API object. Calculating X AND Y gives another value Z. But recall that each bit in Z will either be 0 if the bit in Y was 0, or it will be the bit of X itself.

Can you see the subtle implication from this? If the n'th bit of our filtering value Y was 0 then the n'th bit of the result Z will be 0. There is no loss of information because we didn't care if the bit value in X signaled support for this feature or not. But if the n'th bit of Y was 1 then we also want the n'th bit of Z to be 1. This is only possible if the corresponding bit in X was 1, hence signaling support for this feature. We can thus compare the result Z to Y itself. If X AND Y == Y, then and only then will the value of X signal for (at least) all the features we are filtering for with the value Y.

Let's take a quick example using Unix permission values again. Let's say we have a list of files with permission values and want to filter them to keep only those that we are allowed to read, which is the value 4 or in binary 100. One of the files may have a permission value of 5, which in binary is 101 and thus corresponds to read and execute permission.

A filter to see if at least the read permission was set on this file would be checked as follows:

We see that the result on the last line is exactly the same value as our filter. This is just as we expected and we would therefore keep it.

If a file gives us permission to write and execute, but not to read, it is represented as a 3 or in binary 011. Applying our filter:

The result is different from our filter and we therefore immediately know that the filter was not satisfied.

The real beauty comes from applying complex filters with multiple bits turned on. If we would want to filter for e.g. read and execute permissions, the filter would be 101 and just as before we would check if the final AND result would equal this value.

The value Z resulting from using the AND operator on a file's permission and filter value is the intersection of the permissions in the file and the filter.

Bitmask filtering in Python

Using Python 3.6+

Since Python 3.6 a class Flag exists for bit mask operations.

>>> from enum import IntFlag, auto
>>>
>>> class Permissions(IntFlag):
...    EXECUTE = auto()
...    WRITE = auto()
...    READ = auto()
...
>>> class PermissionSet:
...    def __init__(self, *flags):
...        self._permission = Permissions(0)  # Initiate no permissions
...        for flag in flags:
...            self._permission |= Permissions[flag.upper()]
...    def __contains__(self, item):
...        return (self._permission & item._permission) == item._permission

We can use this PermissionSet class to store the permissions of a file as follows:

>>> class File:
...    def __init__(self, name, *, permission):
...        self.name = name
...        self._permissions = PermissionSet(*permission)

Let's use it in action!

>>> # A filter:
>>> permission_filter = PermissionSet("read", "execute")
>>>
>>> # The dummy files:
>>> files = []
>>> files.append(File("file_rw", permission=("read", "write")))              # Should be excluded from filter
>>> files.append(File("file_re", permission=("read", "execute")))            # Should be included from filter
>>> files.append(File("file_w", permission=("read", "execute")))             # Should be included from filter
>>> files.append(File("file_we", permission=("write", "execute")))           # Should be excluded from filter
>>> files.append(File("file_rwe", permission=("read", "write", "execute")))  # Should be included from filter
>>>
>>> # Filter the files:
>>> for f in filter(lambda f: permission_filter in f._permissions, files):
...    print(f.name)
...
'file_re'
'file_w'
'file_rwe'

Using Python 3.5

However, also before Python 3.6 this could be achieved using the IntEnum class, however with manual adjustment of the underlying permission values.

>>> from enum import IntEnum
>>>
>>> class Permissions(IntEnum):
...    EXECUTE = 1 << 0
...    WRITE = 1 << 1
...    READ = 1 << 2
...
>>> class PermissionSet:
...    def __init__(self, *flags):
...        self._permission = 0  # Initiate no permissions
...        for flag in flags:
...            self._permission |= Permissions[flag.upper()]
...    def __contains__(self, item):
...        return (self._permission & item._permission) == item._permission

We can then use it in the exact same way we already did above in the example for IntFlag.

What do you think about this technique? Is it elegant? Or over-engineering?

Making a Blog With Hugo And Netlify

Bjarne Magnussen — Mon, 02 Sep 2019 21:41:50 +0000

Numerous blog posts already exist to explain how to build a blog with Hugo and deploy it using Netlify. But since this is my first blog post (a warm welcome by the way ❤️) I wanted to briefly describe how I created my own site, which you can find on https://bjarnemagnussen.netlify.com.

I will also clarify things I have often found lacking in other descriptions, such as what to be aware of when using Netlify, how to use a private repository for the raw files and how to use the Academic theme of Hugo. Some prior knowledge of git and GitHub is required.

When I started out to create my own site I wanted something that was easy to deploy and with support for markdown. It is my preferred way to write and the choice quickly went for a static site generator. I chose Hugo because it is written in Go (the language I currently study) and I liked the look of the Academic theme. Easy deployment with Netlify was also a clear plus.

Installing Hugo With Academic Theme

I chose an installation method that uses git and GitHub as described here. However, I didn't want my raw markdown documents and configuration files exposed in a public repository and changed some steps in the installation process.

First create a private repository on GitHub, you can name it whatever you want.
Then clone the academic-kickstart repository to your local machine in a temporary dictionary mirror.

mkdir mirror
cd mirror
git clone --bare https://github.com/sourcethemes/academic-kickstart.git

Finally push it to your private repository created in step 1 (change yourname and private-repo accordingly below).

cd academic-kickstart.git
git push --mirror https://github.com/yourname/private-repo.git

You now have a basic Academic theme skeleton inside your repository and can delete the temporary mirror folder that was only created for this purpose.

Use your repository in any of the usual ways you find convenient. Academic comes pre-installed with many example pages and configuration options. You can find the documentation page for it here and change the example files according to your taste and need.

Deploying The Site

To deploy the webpage on Netlify you will first have to register with them here.

It is completely fine to register using only your email 😉. No need for GitHub's single-sign-on at this point!¹

We can then deploy a new site by pressing the little "New site from Git" button in the top right.

From the new window we select the "Github" option under "Continuous Deployment".

We must then grand Netlify some rights and access to our private repository that we have specifically created for this purpose.

Confirm it by clicking on the repository from within Netlify's window.

And press the "Deploy site" button to deploy using default parameters for hugo.

🎉 Congratulations! Your blog should show up under "Sites" and every new commit to your private repository is automatically deployed.

You can now change the settings for "the-complicated-looking-and-automatically-assigned" subdomain, or switch to using your own domain name. All of that is explained on their documentation page.

Although we must link our GitHub account with Netlify, I always avoid using single-sign-ons from specific vendors. I find it disturbing to use as it complicates parting with the vendors later if we should not use their services anymore... ↩