How to simply get your dependencies with Nix

If you haven't hear of Nix yet - that's perfect! If you have looked into it and came off it a bit bewildered... I understand. Nix can seem very unusual at first, but it can also be of tremendous value for your software projects. I hope we can start fresh for this post. 🙂

Managing dependencies and creating reproducible environments is a Very Hard™ problem

Managing dependencies in software development can be a major pain, and we created a plethora of solutions for it: We have apt, npm, brew, pip, gem, rpm, just to name a few... We also use containers like Docker to manage software packages and create relatively reproducible environments.

All of those solutions have their limitations. Either they are specific to some language (pip for Python, gem for Ruby, stack for Haskell...), to some OS (apt for Debian/Ubuntu, rpm for Fedora & co. ...), to some part of our stack (backend/frontend), or they come with extra complexity (containers). They can be more or less reliable (npm), their results are more or less reproducible. Few solutions are able to sensibly manage multiple versions of the same package.

The solution to all dependency management pains

The solution is obvious: We need another package manager to rule them all!

Nix is getting remarkably close to being an ideal cross-OS, cross-language, cross-stack package manager, while being extremely reliable and reproducible. Of course, it's not perfect and it has its own set of trade-offs and limitations. There is a lot to explain on Nix, how there is a functional programming language called Nix, a package repository called Nixpkgs, a whole operating system called NixOS... and a lot more. But for this post, I would like to show you a simple, minimal use-case that might already be very valuable to you!

Simple example: Managing development dependencies with Nix

Let's say that for hacking on one of our projects we need curl, jq, entr and pg_tmp. pg_tmp in turn depends on the Postgres binaries. This is how to get all that with Nix:

# Install Nix
curl https://nixos.org/nix/install | sh

# Create a `shell.nix` file
cat > shell.nix << EOF
let
  pkgs = import <nixpkgs> {};
in
pkgs.stdenv.mkDerivation {
  name = "my-env";

  buildInputs =
    [
      pkgs.curl
      pkgs.jq
      pkgs.entr
      pkgs.ephemeralpg
    ];
}
EOF

# Run `nix-shell`
nix-shell

# Have all dependencies that we need ready to go on your $PATH! 🎉

That's it!

Installing Nix as above is a relatively invasive change to your system, for example you will find a new '/nix/' directory in your root file system. I would recommend to try this out in a virtual machine or Docker container first.

I guess it makes sense for me to explain a little bit more on what is happening here and how you can use it.

Parts of the Nix ecosystem that we use in this example

We are using three parts of the Nix ecosystem.

1/3: The Nix programming language

It's a very small language, the syntax can be described on one page. If you are familiar with functional programming languages (especially from the ML family, Haskell, Elm), you will be right at home. If you are not, it might seem quite alien and restrictive (no loops?!).

Everything in the Nix language is an expression, that is everything you write evaluates to a value. The language is mostly pure: Most function calls have no side effects and will always return the same result for the same inputs. The remaining impurities are mostly taken care of with cryptographic hashes. Based on this, the results of any Nix expression are highly reproducible.

2/3: The Nix packages repository

Nixpkgs is one huge, but well organized Nix expression that defines how over 40,000 software packages can be built, including all their dependencies. It's currently over 17 mb in total.

The Nix programming language keeps working with this huge expression efficient by being lazy and only evaluating the parts of it that you currently need.

Nixpkgs is where a lot of the complexity that you might encounter with Nix comes from. Abstractions like mkDerivation, callPackage etc. are great for taking out repetition and keeping Nixpkgs maintainable, but their abstractions can be quite difficult to understand. Thinking too much about some of them makes my head hurt.

3/3: The nix-shell utility

The nix-shell binary loads a Nix expression from a file (by default from the files shell.nix or, if that one doesn't exist, default.nix), evaluates that expression and then drops us in a shell where all dependencies defined in the expression are (magically) available.

More explanations and details

With that said, let's go through the example above in more detail.

The shell.nix file explained

Let's have another look at the shell.nix file that we created earlier:

let
  pkgs = import <nixpkgs> {};
in
pkgs.stdenv.mkDerivation {
  name = "my-env";

  buildInputs =
    [
      pkgs.curl
      pkgs.jq
      pkgs.entr
      pkgs.ephemeralpg
    ];
}

The let [DEFINITIONS] in [EXPRESSION] expression allows us to define variables in a local scope in the first part after let, and then evaluates the second part after in with that scope. Accordingly, the variable pkgs defined in the first part can be used in the second part.

import <nixpkgs> {} on the second line is how we import the Nixpkgs expression that is available in our Nix installation by default. I'll explain in a separate post how to 'pin' the Nixpkgs to one specific, reproducible version.

pkgs.stdenv.mkDerivation is a complex function that is used all over Nixpkgs to define how different packages can be built. All we need to know about it for now, is that it takes a set ({...}) as an argument and returns a derivation (that is 'something that can be built' in Nix parlance). The build inputs will be able on our path if we run this expression in nix-shell.

With name = "my-env"; we set the name attribute of the set we use as an argument to mkDerivation. It doesn't really matter what we put here.

buildInputs is the most important part: We set it to a list of derivations that we want to have available in our nix-shell. The items of our list are separated by spaces. We simply use attributes from the pkgs value. Remember that pkgs resulted from importing the huge Nixpkgs Nix expression, so it contains all of the over 40,000 packages in its attributes. We just pick out the right ones.

Finding the right attributes on Nixpkgs is not always simple, but a bit of googling goes a long way. For example, Google told me that pg_tmp is part of the ephemeralpg package in Nix.

The nix-shell utility explained

A lot of wonderful magic happens when we run nix-shell. Let's go through the most significant steps it performs:

1. nix-shell will look for a shell.nix file in the current directory, and immediately find it in our example.
2. It evaluates the Nix expression contained in the file. In most cases, the expression is quite huge as the whole Nixpkgs expression with over 40,000 packages is being imported. Keep in mind that Nix keeps this reasonably efficient by only evaluating the parts that we actually need.
3. Now that the Nix expression is evaluated, nix-shell knows exactly which dependencies are needed. It checks in the local cache located in the /nix/store directory, if the individual dependencies are already built. If not, it will try to get them from a binary cache at cache.nixos.org. Only as a last resort will it actually build the required packages itself.
4. When all dependencies are successfully cached in /nix/store, nix-shell puts together a $PATH environment variable that ties all the requested dependencies together. It then launches a new shell with that path set.

If you only want to run a single command instead of being dropped into a shell, you can use the --run flag. For example, nix-shell --run "curl http://google.com/" will query Google using the curl version that was installed by Nix.

As soon as you leave the Nix shell again with Ctrl-D, any of the changes that nix-shell made to our environment will disappear. It will look like the entr package that we defined as a dependency was never installed, with exception of the cached binary in /nix/store. The latter can be cleaned up with nix-collect-garbage or kept for a faster start up of nix-shell the next time.

More to come

Congratulations on surviving your first toe-dipping with Nix! With the example above, you already have a usable first piece that can easily be extended with additional packages. In an upcoming post, I'd like to show you how to pin your version of Nixpkgs in order to make your dependencies reproducible across time and different systems!

Comments

[Jatin Sharma]

Nice and on the point article mate.

[Tomo]

Niv is also pretty cool for this; it uses Nix Flakes so your dependencies get pinned, and it's a nice and easy-to-use tool.