Flake my life - how do nix flakes work?

So I’m watching the excellent video Jake Hamilton "NixOS Flake tour" because I don't really understand how flakes work. Conceptually it seems clear but there is strange deepness to it too. What are flakes? Are flakes a natural extension or progression of nix expressions? I'd like to setup my system with flakes but it all seems foggy and unclear.

I used to have my NixOS setup with a tiling manager and various configs I copied from other experienced NixOS users but all that was done with a configuration.nix and home-manager and cachix as well. And then there was this really nice xmonad setup by Gabriel Volpe I managed to reproduce and which is a flake too, but after analyzing it and studying all the modules I still haven't actually understand how flakes work. His setup is at github link. Then came the time when I didn't use NixOS and in the mean time flakes have taken over the world it seems. Now I have just the plain plasma DE and simply wish to setup a basic flake system with a home-manager and one output machine. But so far it seems I cannot understand how flakes actually work. I have just applied for Summer of Nix 2023 and although I am incredibly excited about it and wish to do my best if it happens for me, I am almost feeling depressed since I can't seem to understand even how flakes work. How will I jump into the ocean of Nix expressions then?

So Jake very quickly mentions the snowfall-lib he wrote, a project of his. What does that mean? Is the snowflake-lib a library or a program or just another nix expression that would somehow automate flake related stuff? Jake says it “kind of does the boring work of wiring things together” so I guess I will understand more soon. There is a reason he has the folders organized the way they are setup and maybe the snowfall-lib does that for him or the folders need to be setup like that in advance?

.. because those are conventions from snowfall-lib which let’s me just say hey here is my directory with all of my stuff in it and it will import everything and hook it all up. I’m actually using the slightly more complex method here. I could switch this out to snowfall lib.makeFlake but once upon a time I needed access to this lib here. So snowfall-lib, go check it out..

I have no idea what this means. But I'm sure it's just some context beyond my level of understanding for now. I'm still a NixOS newbie.

So I go now to snowfall-lib on snowfall homepage on github and here we see that snowflake is built on top of flake-utils-plus and that snowflake is described as Unified configuration for systems, packages, modules, shells, templates, and more with Nix Flakes.

Oh I see. It seems without this snowfall library two or more users creating different flakes might end up with creating different set of folders or directories in the flake directory. So this means although flakes are reproducible they themselves appear to be impure because of different sets of directories? This is only important for the snowfall library for now since the library can potentially process all the flakes and automate stuff only if the flakes would respect the standard way in which flake directories are created. But since there is no standard way flakes directories are created snowfall enforces that which in turn enables the library to do the automation work. This is so far how I understand how this works.

Oh, now I see he uses deploy-rs for deployment which means this is another tool for automating nix flows. So now conceptually we have snowflake-lib, which is built on top of flake-utils-plus that uses the deploy-rs tool for deployment of outputs? Is deploy-rs something like NixOps? Now why would we need a deployment tool? I guess I will read now the serokell blog post on their tool delpoy-rs:

But reading it I understand that I should go out of this rabbit hole and try to find something on flakes. And in the article the often mentioned article from Tweag is referenced which is at https://www.tweag.io/blog/2020-06-25-eval-cache/

In the past, you might use the NIX_PATH environment variable to allow your project to find Nixpkgs. In the world of flakes, this is no longer allowed: flakes have to declare their dependencies explicitly,
and these dependencies have to be locked to specific revisions.

So this makes sense actually. But now this brings us to cache invalidation which is a serious problem in computer science! The cache itself is not the problem but the stale cache data that is not needed anymore and the problem of updating the cache field. So while on one hand you could update it too much and create much latency you could update it rarely and thus create stale unused data which might bring other problems in the system. This does seem like a hard problem. And this is where apparently flakes come to rescue.

Flakes bring what is called hermetic evaluation into the game, hermetic meaning completely sealed of from external influences. I ask now chatgpt4 to explain me the concept of hermetic evaluation in terms of nix flakes and it does a good job of it.

Chatgpt explains:

Hermetic evaluation in terms of Nix Flakes has the following implications:

1. Explicit dependencies: Nix Flakes forces you to declare all dependencies explicitly, making it clear which external resources are required for the build or evaluation process. This helps ensure that the build process is self-contained and reproducible.

2. Content-addressed storage: Nix Flakes uses content-addressed storage, which means that the outputs of a build are identified by the hash of their content. This ensures that if the inputs to a build haven't changed, the output will remain the same, regardless of external factors.

3. Isolated build environments: Nix creates isolated build environments for each package, with precisely defined inputs and no access to external resources. This ensures that the build process is not influenced by factors outside of the declared dependencies.

4. Lock files: Nix Flakes automatically generates lock files that "pin" the exact versions of dependencies used in a project. This ensures that the build remains reproducible, even when the upstream dependencies change.

5. Composability: Nix Flakes enables you to easily compose different projects together, ensuring that the dependencies between them are well-defined and isolated. This helps maintain the hermetic nature of the evaluation process when combining different Nix-based projects.

Actually now my understanding is that flakes are a natural extension of the nix system. It's like nix itself built on top of nix. The very reproducible nature of nix enables something such as flakes to work. So it's not even a question of inventing flakes, it is a question of finding them, this new solution that solves reproducibility of entire nix package configurations that have an arbitrary number of inputs. So now a flake can have more information cached in a way which will result in a faster evaluation time.

Since the flake’s lock file contains content hashes of all dependencies, this is enough to efficiently and completely capture all files that might influence the evaluation result. (In the future, we’ll optimise this a bit more: for example, if the flake is a Git repository, we can simply use the Git revision as the cache name.

Yeah, so in part three of the series on Nix Flakes the theme is setting up individual NixOS configurations as flakes. One of the things I didn't first like about home-manager is that it was somehow separated from the NixOS configuration but then there was a possibility to include home-manager as a standalone NixOS module and include it in the configuration.nix so that when one runs nixos-rebuild switch it would update the home-manager tree as well. But now when thinking of the flake view on this it does show that let's say I run my setup in a month or so what will happen? My nixos-rebuild switch will pull out a different git revision of packages from the nixpkgs tree. Does that mean it will not work? No, on the contrary, it will work but those will be newer versions of my packages. So now we have two points in time which does make the situation impure. The benefit of flakes is that our system configuration is fully reproducible meaning the packages and the nixpkgs are pinned down to their respective versions at the time when flake.lock is created. At least this is how I understand it now.

As the article goes let's check our NixOS version right now:

❯ nixos-version --json | jq -r .nixpkgsRevision
The program 'jq' is not in your PATH. You can make it available in an
ephemeral shell by typing:
  nix-shell -p jq

~/.config 
❯ nix-shell -p jq
this path will be fetched (0.00 MiB download, 0.01 MiB unpacked):
  /nix/store/65kn5z9zx4vwpib1v4b2kvc8mbalyrvj-jq-1.6-dev
copying path '/nix/store/65kn5z9zx4vwpib1v4b2kvc8mbalyrvj-jq-1.6-dev' from 'https://cache.nixos.org'...

~/.config via ❄  impure (shell) 
❯ nixos-version --json | jq -r .nixpkgsRevision
d9f759f2ea8d265d974a6e1259bd510ac5844c5d

So there is a unique number but you apparently can't extract the configuration.nix or any other modules file from this number.

Oh wow, and now a really cool quote comes:

It’s worth noting that any NixOS system configuration already violates the monorepo assumption: your system’s configuration.nix is not part of the nixpkgs repository.

I have never thought about this but it does make perfect sense. If my configuration.nix is not in a common repo this means it is impure and harder to use!

So another quote nicely outlines what flakes solve:

1. Reproducibility: the entire system configuration (including everything it depends on) is captured by the flake and its lock file. So if two people check out the same Git revision of a flake and build it, they should get the same result.

2. Traceability: nixos-version prints the Git revision of the top-level configuration flake, not its nixpkgs input.

3. Composability: it’s easy to pull in packages and modules from other repositories as flake inputs.

OK so here we go. Now I already am on NixOS unstable, so I just need to enable the flake enabling option in my configuration.nix but here is where the first problem arises because I am not sure if this relates to a pure NixOS system or a system that has just the Nix package manager installed:

nix.extraOptions = ''
  experimental-features = nix-command flakes
'';

The thing is that the NixOS wiki on Flakes states a different nix setting for the configuration.nix:

{ pkgs, ... }: {
  nix.settings.experimental-features = [ "nix-command" "flakes" ];
}

But then I stumble on this recommendation on NixOS discourse:

nix = {
   package = pkgs.nixFlakes;
   extraOptions = lib.optionalString (config.nix.package == pkgs.nixFlakes)
     "experimental-features = nix-command flakes";
};

Now yesterday I watched also excellent three hour video on setting up NixOS at NixOS Setup Guide from which I did:

nix = {
  package = pkgs.nixFlakes;
  extraOptions = "experimental-features = nix-command flakes";
};

Now what is going on here and why are these expressions different and yet apparently doing the same thing. Now we come to nix programming! I asked chatgpt to explain me the differences between all these expressions and after some chatting back and forth it is evident now:

• that the first expression which was shared on NixOS discourse is actually conditionally enabling experimental features only in the case of flakes

• the last expression actually enables experimental features in whatever case so not just for flakes.

How is the first expression conditionally enabling flakes is the cool part! With the lib.optionalString which is a function provided by the nix library.

The syntax is like:

lib.optionalString condition stringValue

In the first expression:

extraOptions = lib.optionalString (config.nix.package == pkgs.nixFlakes)
     "experimental-features = nix-command flakes";

the value of extraOptions is conditionally set to "experimental-features = nix-command flakes" only if config.nix.package is equal to pkgs.nixFlakes. This means that the "experimental-features" option will be included only when the system is using the nixFlakes package.

Ok so after seeing more chatgpt examples of lib.optionalString utility I see that it's a really practical way to include flags into builds and configurations that might or might not happen depending on some conditions.

This flake my life will need to go on. My goal is to make the simplest possible flake of my system and make sort of a spectrum of possible ways how the flake can expand from a simple to more complex system. I often find out complex flake configurations with just enough nix expressions variations and that makes it a bit more difficult for me to understand. But I learned the cool lib.optionalString though which is awesome!