Easy deploy of docker based projects

I have a personal server where I run some projects, some of them written in python, elixir and other technologies so having to deal with specific installation of any of these technologies is not an ideal workflow, to fix this I use docker and all of them are deployed using docker-compose, they're connected to a single PostgreSQL server and they're behind the same web server.

Running all of these projects in this way it's easier to maintain and in case something happens with the server I can re deploy everything in a easy way. Let's take a look to these tools and how they work together.

Let's assume we have the following requirements:

• Deploy a django application
• Deploy a phoenix application
• Each application needs a PostgreSQL database
• Both applications should be behind a web server and being accessed over HTTPS
• All of these should run in the same server

To solve this we're going to:

• Set up a linux server
• Install PostgreSQL
• Configure a web server which will handle incoming traffic and SSL termination
• Run our applications inside Docker containers

Setting up a server

If you already have a server you can skip this section.

We first need a server which can run docker, most linux distros can be used for this but in this case we'll be using Ubuntu Server, if you don't have a server yet you can use any of these referral links to get some credit when you create your account:

• Digital Ocean this will get you $100 on credits to be used in 2 months
• Hetzner this will get you 20€ on credits, this provider has cheaper prices than Digital Ocean but it only have data centers in Europe
• Linode This is not a referral link but you can get $100 on credits, AFAIK it doesn't expire

Once you get a server it's recommended to make some basic configurations like updating packages, setup a firewall, etc. You can follow this Linode guide to secure your server.

After that you need to install docker, to do that you can follow the official documentation, this have specific instructions for you linux distribution.

Once we have docker and docker-compose installed we can follow this guide.

Installing PostgreSQL in our host machine

We're going to use a unique instance of PostgreSQL installed in the host machine, this way we can share the resources used by PostgreSQL with all the applications that we're going to deploy, we just need to create new users and databases for each one of the applications.

First let's install PostgreSQL with:

sudo apt install postgresql-server

We need to login with postgres user so we can be able to enter to a psql session. We can do it with:

sudo su - postgres

Now we can open a psql session and create the databases and users for our applications:

postgres=# CREATE USER django WITH ENCRYPTED PASSWORD 'secret';

postgres=# CREATE DATABASE django WITH OWNER django;

And we do the same for our phoenix application:

postgres=# CREATE USER phoenix WITH ENCRYPTED PASSWORD 'secret';

postgres=# CREATE DATABASE phoenix WITH OWNER phoenix;

Configuring Caddy as a reverse proxy

Caddy is a "new" web server written in Go that have 2 main features that make it a good option for simpler deployments:

• Simpler configuration file
• Free auto configured SSL certificates, using Let's Encrypt service, and automatic renewals

If we were using for example Nginx we have to deal with HTTPS certificates by ourselves, installing certbot, and also have to configure some way to renew the certificates, Let's Encrypt issues certificates that expire after 3 months.

Let's define our domains django.domain.com and phoenix.domain.com which will send traffic to their specific applications.

Our django application needs that Caddy serves the static files so we define file_server option and tell caddy where are our static files, we also tell Caddy to send the traffic to port 8000 where our application is listening.

django.domain.com {
    root * /opt/django

    @notStatic {
        not path /static/*
    }

    reverse_proxy @notStatic localhost:8000
    file_server
}

Our phoenix application will serve static files by itself so we just need to define the reverse_proxy directive to be able to send the traffic to port 4000

phoenix.domain.com {
    reverse_proxy localhost:4000
}

Now when we reload our caddy server with sudo systemctl reload caddy it will get the SSL certificates and internally will check if they still valid, otherwise it will renew them.

Running our projects with docker-compose

Docker compose is a tool that allow us to define different docker services in a easier way using a yaml file.

We're going to configure our two projects using docker-compose but we first need their docker images so let's build them.

Let's clone our projects(both are in the same repository, just in different folders), build the images and then publish them on a registry.

This can be made in a separate machine because once the images are pushed to a remote registry they can be downloaded in our server.

cd simple-django-project-with-docker
docker build -t registry.mycompany.com/django:v1 .
docker push registry.mycompany.com/django:v1

cd simple-phoenix-project-with-docker
docker build -t registry.mycompany.com/phoenix:v1 .
docker push registry.mycompany.com/phoenix:v1

You can use docker hub to push your images or use Gitlab registry in case you want free private images.

Django application

Let's create a folder in /opt/django and put the following code into a docker-compose.yml file.

version: "2"
services:
  web:
    image: registry.mycompany.com/django:v1
    restart: always
    network_mode: host
    environment:
      ALLOWED_HOSTS: "django.domain.com"
      DEBUG: "0"
      DATABASE_URL: "postgres://django:secret@localhost:5432/django"
      DJANGO_SETTINGS_MODULE: "config.settings"
      SECRET_KEY: "a 32 long secret key"
    volumes:
      - ./static:/app/static
    ports:
      - "127.0.0.1:8000:8000"

The static folder will be used by Caddy to server static files.

Phoenix application

Now for our phoenix application let's create a folder /opt/phoenix and put the following code into a docker-compose.yml file.

version: "2"
services:
  web:
    image: registry.mycompany.com/phoenix:v1
    restart: always
    network_mode: host
    environment:
      DATABASE_URL: "postgres://phoenix:secret@localhost:5432/phoenix"
      MIX_ENV: prod
      HOST: "phoenix.domain.com"
      SECRET_KEY_BASE: "a 32 long secret key"
    ports:
      - "127.0.0.1:4000:4000"

Because we're running PostgreSQL in our host machine instead of a docker container we have to use network_mode: host, this allow us to access postgres just pointing to localhost.

Deploying our projects

Once we have the docker-compose.yml files configured we can go inside each project folder and run:

docker-compose up -d

For the django application we also have to run these commands, these are specific of django deployment process.

# Run database migrations
docker-compose exec -T web python manage.py migrate

# Collect all static files and place them in our STATIC_ROOT folder which will be served by Caddy
docker-compose exec -T web python manage.py collectstatic --no-input

Deploying new changes

Because we're using docker, when we need to update changes we just need to update their Docker images and restart their services. Some technologies can have differences in their deployment process but the basic idea is the same.

Let's see how it could be for our two example applications.

Django application

When we update a django application we need to run some extra commands like migrate, collectstatic, etc. We can follow these steps to run them inside the docker container:

docker pull NEW_DJANGO_IMAGE

sed -i "s/image.*/image:\ NEW_DJANGO_IMAGE/" docker-compose.yml

docker-compose up -d --force-recreate

docker-compose exec -T web python manage.py migrate

docker-compose exec -T web python manage.py collectstatic --no-input

We're pulling the new image from our registry, updating the image value in our docker-compose.yml file, restart the service (it will use the new image now) and then we can execute migrate and collectstatic commands

Phoenix application

For the phoenix application we're going to follow almost the same process with just one difference, we don't need to run migrations in a separate step because they will run when the application starts, this is defined in the phoenix docker image itself.

So we just need to pull the new image, update it in docker-compose.yml file and then restart the service, the final script will be:

docker pull NEW_PHOENIX_IMAGE

sed -i "s/image.*/image:\ NEW_PHOENIX_IMAGE/" docker-compose.yml

docker-compose up -d --force-recreate

Conclusion

Having a central PostgreSQL instance and a central web server(Caddy), both in the host machine instead of inside a container allow us to manage them easily and also allow us to share these common services alongside the many applications that we are running in our server.