Mono repository pattern in Elixir

Hello everyone! My name is Alexander Ulyev, I am an Erlang/Elixir enthusiast.

Recently while building a chat application, I ran into a situation when I felt like my application needs to nest other applications in it. Mix tool gives a solution for that - umbrella applications. Great! I was happy with that until I discovered another umbrella application inside my project. There's no out-of-the-box solution for that case, so I stopped working on my chat app and switched to reading mix's documentation and source codes. In the end I developed a pattern for building complex applications and today I want to share it with you. There will be some useful insights about mix application in general as well as mix tasks and tuning MixProject configuration along the way. Let's dive in!

Going beyond umbrella application

Imagine you've got a bunch of applications that don't make sense without each other, yet must be separated to satisfy design requirements. Perhaps you want to run one of those apps on separate or/and multiple nodes or it's just a feeling that the app will have it's own life. And then it gets children apps itself! Or even more! You can end up having applications deeply nested in your main app. Should we avoid that? If apps are related, then they are related and must be developed together. Breaking designs and concepts in favour of ease isn't a good thing, so let's find a proper solution for the case.

While reading umbrella applications documentation, I came across a term - "mono repository". It sounded like a solution for my case, but there were no ready-to-use modules and tasks for convenient configuring and managing such a repository. I'm not a member of mix core team, so I decided to go through all related mix's features to find ways of solving my problem without changing the way mix works. Let's see what mix has to offer!

Mix

First of all, let's talk a bit about umbrella applications. I'll skip basic info as it can be easily found in mix help new output.

First question I had to answer was: what makes umbrella application an umbrella? :apps_path - that's it. If there's such a key with a string (can be empty) value, the project is considered to be umbrella and all applications in :apps_path - to be it's dependencies.

A word on dependencies. Dependencies can be conveniently compiled and released with single line each, just like any application. Later we'll make a use of that. Watch out: you can't define a dependency for your umbrella app if the dependency is in your :apps_path folder. This results in an unresolvable dependency definition conflict, which can't be fixed with :override key (not sure if it's a bug or a feature).

Umbrella project ignores any test task commands as it is not supposed to (but can) have any source code files, therefore - tests... Until you comment out that line and add :app key.

Discovering scopes

Imagine we've got a following applications' tree:

app
   /app0
   /app1
        /app00

app will be our main project - the root, app0 and app1 - tier 1 children and one tier 2 child - app00. Root, parent, child, leaf - tree structures terminology will be used further in this article.

Looking at this structure, let's think:

• how can we test all applications in the tree at once?
• how can we keep all dependencies source, build and lock files at one place?
• how can we make a release of any set of those apps?

And we must consider application tree of any complexity.

Mix projects tend to define a single configuration for most of project's needs. This approach suits well until project starts getting complex. Just like with any other task - when it starts getting complicated, we break it in smaller pieces, focusing and processing them one by one. Let's follow that and break the idea of mix project into mix project scopes: building, testing and releasing an application.

Testing

Of course individual applications get tested during their development process (assuming TDD). However it can be a good idea to test a subtree or entire tree of applications before making a release. There are things to take care of to make it work:

1. You must have all the configuration required by all the applications in a single file mapped on :config_path key in your root or parent application's project declaration.
2. You must start all applications before running tests. This can be achieved by declaring your children apps to be your dependencies through :deps key.
3. :test_paths key must be mapped to a list of paths of your applications' test_helper.exs (test folder by default).

And again, the presence of :apps_path key denies all test task commands, absence of :app key raises an exception.

Building

It makes sense to keep all dependencies source and built beam files at one place, so we don't duplicate any of them. And it makes sense to keep them all at the application root's folder.

mix.lock file tracks fetching dependencies, and it is checked with by mix every time you issue mix deps.get command. The default path for lock file is ".", but we'll change it using :lockfile key. :deps_path points to dependencies source codes path, :build_path - to built beam files, :config_path - to application's configuration. I find it more convenient to have application's configuration at the same folder with application instead of having a single cumbersome file for all children apps. If you think opposite, you can map all four mentioned keys to the main application root's folder. If not - just three of those.

Releasing

Both the main application and it's dependencies get released in the same way, so it's not that complicated to setup a proper project's configuration. However, release scope is too different from testing and building, so we must use a different mix.exs and pass a path to it through MIX_EXS environment variable. Since there are more than one mix project files, we must take care of :version value - it must be same in both files. So, what must be configured for a release of mono repository?

1. :deps must be a list of applications to release.
2. :config_path must be a path to release-related configuration file. 3. :releases key to be defined as instructed in official documentation

It makes sense to have an individual configuration for each release instead of pushing unrelated parameters to various sets of applications.

mono_repo

For now we have talked about all the configuration we need to setup in order to keep our repository testable, releasable and free from duplications. It can be tedious and error-prone to define all the paths (that mostly consist of dots and slashes) and dependencies in every project's mix.exs. That's why I developed mono_repo - a library that removes necessity of literal definitions and substitutes them with function calls. The library is split in three modules named after scopes they focus: Test, Build and Release. I named functions after the keys they are meant to be used for, prefixed with build_. For example, :config_path can be defined with build_config_path(). Let's make a short stop-by at every module.

MonoRepo.Test

The main task here is to build test folders paths and dependencies lists, but you can also use build_config_files/0 to let MonoRepo assemble all children's config.exs and test.exs into corresponding files in your root's config folder. The module's functions do not require arguments. To be used in root's/parent's mix.exs.

MonoRepo.Build

Using this module's functions saves time and nerves by building paths all the way up from a child to root/parent. Zero-arity functions assume the target parent to be root, one-arity functions need a target application name as an argument. To be used in children's mix.exs.

MonoRepo.Release

This module needs some configuration in order to do it's job. A guide can be found in MonoRepo and MonoRepo.Release documentation. As soon as you setup the configuration, the rest will be taken care of this module's functions.

Epilogue

That's all we need to setup a mono repository in Elixir. As we saw, it takes some efforts, but it saves time afterwards. And, in case of using mono_repo library, adds couple of new features like testing entire application tree and building cleaner releases.

I found it interesting to focus mix scopes and what it takes to make mix work the way you need it to. If you want to learn more about mix, I recommend reading it's source codes and using MIX_DEBUG environment variable set to true.

Thank you for reading!