Embedded Elixir (without Nerves)

This story was originally posted by me on Medium on 2018-08-29, but moved here since I'm closing my Medium account.

I love Nerves. Unfortunately I am in a situation where I can not use it for my current project.

This is a somewhat detailed guide on how to run Elixir on an embedded linux without using Nerves.

My use case

I am a co-founder and a backender of Wise Home, a company located in Denmark. We do smart-home stuff for rental buildings. Our hardware pusher provides a small linux-box, which we call a gateway.

The gateway comes with some software used to communicate with external devices. This is provided as a modified buildroot with the hardware pusher’s custom patches and software. Buildroot is a tool that can build linux images, mostly used for embedded devices.

I can put my extra software on the gateway by modifiying that buildroot configuration, but it would be hard (maybe impossible?) to put their software and patches on my own buildroot config, which is why I can not use Nerves.

How to add Erlang to a buildroot configuration?

The linux image that is the output of running buildroot must contain everything needed to run the gateway. This is because most of the file system is read-only, because the gateway should not be able to get to a state where it can’t just reboot and everything is fine again.

This means that we have to put Erlang on the linux image. The gateway runs an ARM processor, so we must cross-compile Erlang to ARM. The heavy work of this is done by buildroot, so we can just configure it to install Erlang.

The buildroot configuration I’m locked to is outdated and contains an old Erlang config, but luckily I was able to just pull the latest Erlang config into my own buildroot without any trouble.

From here it is just a matter of selecting Erlang when configuring buildroot:

Buildroot will now do the heavy lifting of actually cross-compiling Erlang and add it to my linux image.

If you have never used buildroot before and want to use it, I recommend spending a little time on their guide.

So how is Elixir installed, then?

I thought I needed to also do some cross-compiling for Elixir, but I had two eureka-moments when investigating this.

First of, Elixir does not have any native executables. An Elixir installation is basically a bunch of bash-scripts and some beam-files. The bash-scripts starts Erlang and the beam-files are Elixir compiled into BEAM-code.

Secondly, Elixir (the beam files compiled from Elixir) is already included when making a release with Distillery. It is actually “just” a library sitting there with all the other included libraries:

This means that if we can deploy a Distillery release of our app onto the linux image, we’re golden. So that’s up next.

Deploying a Distillery release to the linux image

The only non-default configuration I made to my Distillery build was to not include ERTS:

release :balrog do
  set version: current_version(:balrog)
  set include_erts: false
  set include_src: false
end

ERTS is the Erlang runtime (Erlang Run-Time System) and that is what we needed to cross compile. When not including ERTS in the release, we must make sure that the Erlang version used to build the release is exactly the same as on the target linux image.

Normally when I “deploy” stuff it means that I build a release which is uploaded to a server. With embedded devices, we need to include it in the linux image before it is booting up for the first time.

With buildroot, we can simply add files to the file system of the linux box. This is called “rootfs overlay”. For convenience, I made a small script that builds the release and unpacks the resulting tar onto my rootfs overlay.

After the next buildroot build, it will be contained in my linux image.

Running the release on a read-only file system

By now, I thought I was done. But no. The Distillery release expects by default the file system to be read-write, but on an embedded linux, most of the file system is read-only.

The first attempt to run the release resulted in an error message saying that the log file folder could not be made.

It turns out that a lot of environment variables can affect how a release is started. I ended up using these four in a winning combo:

PIDFILE: On the gateway there is a monitoring system called monit. It expects each monitored application to write its own process-id to a file. With PIDFILE we tell Distillery where that file should be located.

LC_ALL=en_US.UTF-8: Not really Erlang or Elixir specific, but Elixir complained if Erlang was not run with UTF-8. I added the en_US.UTF-8 locale via buildroot and select to use it with this env var.

RELEASE_MUTABLE_DIR: This is the env var that saved my day. I can tell distillery where all mutable files can be located. On my gateway the /data dir was read-write, so I chose a subdir of that.

HOME: Again, not specific to Erlang, but Erlang tries to save a cookie file in the home dir for some reason. Since the home dir in my case was read-only, I set it to something else when running the release.

All in all, this is the command I ended up with that worked nicely:

HOME=/data/balrog/ \
RELEASE_MUTABLE_DIR=/data/balrog/ \
LC_ALL=en_US.UTF-8 \
PIDFILE=/tmp/balrog.pid \
/balrog/bin/balrog start

Yes, my app is called Balrog. Don’t ask. I might do a post later on naming things, which is of course the hardest thing in computer science.

Further reading and investigation

I found a lot of interesting and useful things on my way to deploy Elixir on an embedded linux.

Things I can recommend to investigate:

Buildroot. This is a good tool to know if you do anything on embedded devices. E.g. if you use Nerves and need to modify which applications and libraries are installed on the linux image.

Distillery. It goes without saying that this is an important tool in Elixir-land. But don’t just read the documentation. I found a lot of useful stuff in comments in the codebase.

Your filesystem. I found a lot of clarification by looking at my local Elixir installation and the Distillery release generated from my application. Browse though the directories and peak inside the bash scripts.

Comments

[Paulo Renato]

This is because most of the file system is read-only, because the gateway should not be able to get to a state where it can’t just reboot and everything is fine again.

Sounds like that the use of an almost read-only file system will also improve its security.

So only the /data dir is writable?

[Lasse Skindstad Ebert]

IMHO the greatest advantage of the write-only read-only filesystem on an embedded device is that a restart of the device is more likely to fix a bad state and make the device boot successfully.

Yes, on my specific device, the /data dir was already mounted as read-write.

[Paulo Renato]

I think you meant read-only instead write-only in your reply?

[Lasse Skindstad Ebert]

I did 👍

[Paulo Renato]

HOME: Again, not specific to Erlang, but Erlang tries to save a cookie file in the home dir for some reason.

The reason for this cookie existence is to allow to run distributed Erlang/Elixir. So the cookie will be used to allow nodes to talk with each other.

You can read more here, where at some point you will read:

First of all, you need to ensure all machines have a ~/.erlang.cookie file with exactly the same value.

[Lasse Skindstad Ebert]

Correct :)

You can also set the cookie on startup of the application with some command line argument or env var, which might prevent erlang from saving a generated cookie. I haven't tried this though.