Deploying Elixir and Docker in Docker

Recently I have been building a CI (continuous integration) app in elixir.

Why would I do that when there are plenty of good SaaS options? For Fun!

Along the way I have found out about a few interesting things that I thought I could share. This post will focus solely on Elixir Release and Docker in Docker. I will post another article on the elixir app itself.

What Does the CI App Do?

This is a simple CI application. It waits for Github webhook notifications, downloads and tests the new code, then sends the results back to Github.

Working with github is detailed here: https://developer.github.com/v3/guides/building-a-ci-server/

How Does it Work?

Here is where things start getting tricky. Because we want to isolate the code being tested, we will need to run the tests inside a docker container.

But also for portability of the CI app, our elixir code will also need to be running in a docker container.

This brings us to the ultimate mind bender: Docker in Docker!

Lets jump right in.

Dockerizing an Elixir Application

Best practice to send elixir apps off to production is to use the built in Erlang Releases concept. For elixir apps, this can be easily done through the Distillery library. https://elixirschool.com/en/lessons/libraries/distillery/

One hiccup is that the release needs to be built on an operating system matching the operating system where it will be deployed. Docker can come to the rescue here because it allows you to build your elixir releases inside of docker to guarantee the deployed OS is identical to the build OS.

A good post on the subject is here: https://medium.com/@pentacent/getting-started-with-elixir-docker-982e2a16213c
And the officially recommended Dockerfile example is here: https://hexdocs.pm/distillery/guides/working_with_docker.html

Docker in Elixir in Docker

Remember, we want the OS with our elixir app to include docker so that we can launch containers for testing. The guides above show how to build an elixir release in docker, but not with the OS needed for docker in docker.

That OS is called "Docker in Docker"!

The official image is docker:dind, but there are several variants on dockerhub. We will be using a specific version docker:18.09.0-dind so that we are sure it matches the elixir:1.7.2-alpine OS used to build the release

From the hexdocs.pm example, make a new Dockerfile and change the line
FROM alpine:${ALPINE_VERSION}
to
FROM docker:18.09.0-dind

Dockerfile CMD Bug

Through trial and error I found that the line
CMD trap 'exit' INT; /opt/app/bin/${APP_NAME} foreground
was overriding the CMD from the docker:18.09.0-dind image.
This results in the docker daemon not starting, which makes the image useless.

To fix this I tracked down what the docker:18.09.0-dind CMD is and put it in a shell script.

So in the top level project directory add the following file start.sh
The sleep command is there to allow the docker daemon to start up before the elixir app starts.

#!/bin/sh
dockerd-entrypoint.sh & 
sleep 10
/opt/app/bin/ci foreground

Then update the end of the DockerFile to

COPY ./start.sh .
RUN ["chmod", "+x", "start.sh"]
ENTRYPOINT /start.sh

Inner Docker Image

We've sorted out the container that runs the elixir app on a docker:dind image. Now lets prepare the docker image that the tests will use to run.

Create a folder at test_runner
Within this folder my Dockerfile looks like this

FROM node:11.12.0-alpine
RUN apk update && apk upgrade && apk add --no-cache bash git openssh
COPY . .
RUN ["chmod", "+x", "start.sh"]
ENTRYPOINT /start.sh

and the start.sh file looks like this (with personal details replaced)

#!/bin/sh
git clone --depth 1 -b branch_name https://$[e-mail]/$USER_NAME/$PROJECT.git
cd $PROJECT_NAME
npm install
npm run test

Your inner docker image can be whatever you want. And note it doesn't need any special docker:dind OS to run.

Getting the Image into Your App

So we've made the testing image, but now we need the elixir app to have access to it inside the docker:dind system.

We can do this using the docker save and docker load commands. These allow us to save a docker image to a file and then load it back up.

If we run docker save test_runner > test_runner.tar in the test_runner folder, we will get a new test_runner.tar file.

Then in our main ./Dockerfile we can add the line (right before ENTRYPOINT)
COPY ./test_runner/test_runner.tar .
This brings the file into our Elixir image.

Finally we add this line to the main ./start.sh file
docker load < ./test_runner.tar
This adds the image to our docker application so that is can be fired up from the elixir app.

The final ./start.sh file should look like this

#!/bin/sh
dockerd-entrypoint.sh & 
sleep 10
docker load < ./test_runner.tar
/opt/app/bin/ci foreground

Starting a Container in Elixir

Now that we have set up all the infrastructure for our apps, we are ready to use the container in the elixir app.

Wherever it is needed you can use
System.cmd("docker", ["run", "test_runner"])
That will run the docker image and return you the output and exit status in a tuple.

Don't forget that we can tailor the test runner on the fly by passing in environment variables as well.

Conclusion

Thank you if you've made it this far. No one said docker in docker would be easy!

I realize that elixir deployment methods can be contentious, and docker in docker even more so. I welcome any comments on alternative/better systems we could use here instead.

Now here is a build script I wrote to automate all of the build steps after you set up your elixir project, Dockerfiles, and start.sh scripts.

Final note, you must use the --privileged flag to make use of docker in docker.

#!/bin/sh
cd test_runner
docker build -t test_runner .
docker save test_runner > test_runner.tar
docker image rm test_runner
cd ../
docker build -t elixir_app .
rm test_runner/test_runner.tar
docker run -p $OUTER_PORT:$INNER_PORT --privileged -d elixir_app