Deploying Elixir (3 of 3): Provisioning EC2 With Ansible

This post is the third in a series of three on deploying elixir:

1. Building Releases with Docker & Mix
2. Terraforming an AWS EC2 Instance
3. Deploying Releases with Ansible

---

In pt 2. we installed Terraform and built some infrastructure on AWS, specifically a Debian Buster EC2 instance to run our app. Now that we have our release built, and a place to run our app, let's walk through the basics of Ansible and provision our new EC2 instance.

If you did not follow along with that last post, you can grab the complete code here: https://github.com/jonlunsford/webhook_processor

---

Ansible is:

A radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.

Installing Ansible

If you have pip installed, you can run:

$ pip install --user ansible

Otherwise take a look at their Installation Guide for in depth instructions on how to install on your system. Verify the install worked properly by opening a new shell and typing:

$ ansible --version

You should see some output about the install, mine shows:

ansible 2.8.0
...

Here's the outline of the tasks we will automate with Ansible:

1. Setup: Bootstrapping the new server with all the prerequisites
   • Ensure python is installed (Ansible is a python lib)
   • Create our deploy user/group
   • Create our directory structure
   • Upload various system files (sudoers, systemd)
   • Forward traffic from port 80 to our running app
2. Deploy: Upload our build artifacts
   • Unzip our app tarball in the correct directory
   • Start/restart the app
3. Profit!

Finally, we'll wrap this all up in mix task so we can run:

mix ansible.playbook setup
mix ansible.playbook deploy

---

Configuring Ansible

Create an inventory file noting the public dns of your ec2 instance, this tells Ansible the hosts we would like to run commands on:

# ./rel/ansible/inventory/main.yml

---
all:
  children:
    webservers:
      hosts:
        ec2-x-xxx-x-xx.us-west-1.compute.amazonaws.com

Next, we'll use an ansible.cfg file to store our configuration:

# ./rel/ansible/ansible.cfg

[defaults]
inventory=./inventory/main.yml
remote_user=admin 
private_key_file=../webhook_processor_key 

With these few files in place, we can now test to see if Ansible can connect to our ec2 instance, cd into ./rel/ansible and run:

ansible webservers -m ping

Notice we explicitly called the command with webservers, this matches our inventory file above and scopes our command to only run on those hosts. You could have further inventory like dbservers (or whatever is relevant) and you would scope any commands or playbook runs accordingly.

The last configuration step we will do is create a shared vars file under ./rel/ansible/vars/main.yml. This way we can populate and use common vars across our tasks:

# ./rel/ansible/vars/main.yml

---
mix_env: "{{ lookup('env', 'MIX_ENV')}}"
app_port: "{{ lookup('env', 'APP_PORT') }}"
app_name: "{{ lookup('env', 'APP_NAME') }}"
app_vsn: "{{ lookup('env', 'APP_VSN') }}"
app_local_release_path: "{{ lookup('env', 'APP_LOCAL_RELEASE_PATH') }}"

app_local_path: "{{ app_local_release_path }}/{{ mix_env }}-{{ app_vsn }}.tar.gz"

# Main app server config
# OS user that deploys / owns the release files
deploy_user: deploy
# OS group that deploys / owns the release files
deploy_group: "{{ deploy_user }}"

# Base directory for deploy files
deploy_dir: "/opt/{{ app_name }}"

# Dirs owned by the deploy user
deploy_dirs:
  - "{{ deploy_dir }}"

Notice we'll use the lookup('env', 'MY_ENV_VAR') function to grab a few details about our app, more on that when we write the mix task later.

Setup Tasks

Now that we've verified we can connect, we can begin provisioning our ec2 instance. First, we'll write a playbook that runs various setup tasks:

# ./rel/ansible/tasks/setup.yml

---
- hosts: webservers
  gather_facts: False
  become: yes
  vars_files:
    - "../vars/main.yml"
  tasks:
    # Ensure python is installed, the debian ec2 instance does by default
    # Ansible will show us a warning about that
    - name: Install python 2
      raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)

    # Add the deploy group set in ../vars/main.yml if it does not exists
    - name: "Add {{ deploy_group }} group"
      group: name={{ deploy_group }} state=present

    # Add the deploy user set in ../vars/main.yml if it does not exists
    - name: "Add {{ deploy_user }} user"
      user: name={{ deploy_user }} groups={{ deploy_group }} append=yes state=present

    # Create the directories for the app
    - name: Create deploy dirs
      file: path={{ item }} state=directory owner={{ deploy_user }} group={{ deploy_user }} mode=0700
      with_items: "{{ deploy_dirs }}"

    # Create a sudoers file that gives the app user ONLY the ability to
    # start/stop the app. This helps ensure the security of the server, in case
    # the app user were compromised
    - name: Create sudoers config for deploy user
      template:
        src: "../templates/sudoers.j2"
        dest: /etc/sudoers.d/{{ deploy_user }}-{{ app_name }}
        owner: root
        group: root
        mode: 0600

    # Upload our systemd template
    - name: Copy systemd config file
      template:
        src: "../templates/systemd.j2"
        dest: "/etc/systemd/system/{{ app_name }}.service"
        owner: root
        group: root
        mode: 0644

    # Enable the systemd service
    - name: Enable service
      service: name={{ app_name }} enabled=yes

    # Forward port 80 to our app_port, set in ../vars/main.yml
    - name: "Forward port 80 to {{ app_port }}"
      iptables:
        table: nat
        chain: PREROUTING
        in_interface: eth0
        protocol: tcp
        match: tcp
        destination_port: "80"
        jump: REDIRECT
        to_ports: "{{ app_port }}"
        comment: "Redirect web traffic to port {{ app_port }}"

We're using a couple template files here as well, let's create those next, first the systemd service file that will allow the app to be managed by systemctl:

## ./rel/ansible/templates/systemd.j2

[Unit]
Description={{ app_name }}
After=local-fs.target network.target

[Service]
Type=simple
User={{ deploy_user }}
Group={{ deploy_group }}
WorkingDirectory={{ deploy_dir }}
ExecStart={{ deploy_dir }}/bin/{{ mix_env }} start
ExecStop={{ deploy_dir }}/bin/{{ mix_env }} stop
EnvironmentFile={{ deploy_dir }}/{{ app_name }}.env
LimitNOFILE=65536
UMask=0027
SyslogIdentifier={{ deploy_user }}
Restart=always
RestartSec=5
RemainAfterExit=no

[Install]
WantedBy=multi-user.target

Next, the sudoers file that will only permit the app user (deploy) to stop/start/restart the app:

## ./rel/ansible/templates/sudoers.j2

# {{ ansible_managed }}
{{ deploy_user }} ALL=(ALL) NOPASSWD: /bin/systemctl start {{ app_name }}, /bin/systemctl stop {{ app_name }}, /bin/systemctl restart {{ app_name }}, /bin/systemctl status {{ app_name }}, /bin/systemctl status {{ app_name }}
Defaults:{{ deploy_user }} !requiretty

The last template we'll need is our env file, allowing us to set any vars the app may need. In this case, just the app_port:

## ./rel/ansible/templates/env.j2

APP_PORT={{ app_port }}

---

Ansible Mix Task

Since running these commands will become redundant, let's write yet another Mix task to facilitate everything, including setting all the env vars ./rel/ansible/vars/main.yml is picking up. Create the initial namespace:

# ./lib/mix/tasks/ansible.playbook.ex

defmodule Mix.Tasks.Ansible.Playbook do
  use Mix.Task

  @shortdoc "Run ansible playbooks"
  def run(args) do
    Mix.Task.run("ansible", args)
  end
end

Our task will need to setup a few env vars, then kick off the ansible playbook:

# ./rel/ansible/tasks/ansible.ex

defmodule Mix.Tasks.Ansible do
  use Mix.Task
  use Mix.Tasks.Utils

  @shortdoc "Run ansible playbooks"
  def run([playbook]) do
    cmd_args = ["./rel/ansible/tasks/#{playbook}.yml"]
    {dir, _resp} = System.cmd("pwd", [])
    dir = String.trim(dir)
    mix_env = System.get_env("MIX_ENV")

    System.cmd(
      "ansible-playbook",
      cmd_args,
      env: [
        {"ANSIBLE_CONFIG", "#{dir}/rel/ansible/ansible.cfg"},
        {"APP_NAME", app_name()},
        {"APP_VSN", app_vsn()},
        {"APP_PORT", app_port()},
        {"MIX_ENV", mix_env},
        {"APP_LOCAL_RELEASE_PATH", "#{dir}/_build/#{mix_env}"}
      ],
      into: IO.stream(:stdio, :line)
    )
  end
end

Notice we're setting all the env vars that are relevant to the app, that will become available to Ansible. Run the setup command now, from the project root:

export MIX_ENV=prod
mix ansible.playbook setup

You should see all of the output indicating the result of each task.

---

Deploying the App Tarball

Once setup is complete, the next step is a matter of uploading our build artifacts, here's the entire playbook needed to actually deploy:

# ./rel/ansible/tasks/deploy.yml

---
- hosts: webservers
  become: yes
  vars_files:
    - "../vars/main.yml"
  tasks:
    - name: Copy env to webserver
      template:
        src: "../templates/env.j2"
        dest: "{{ deploy_dir }}/{{ app_name }}.env"

    - name: "Check if {{ deploy_dir }} exists"
      stat:
        path: "{{ deploy_dir }}/{{ app_name }}.tar.gz"
      register: new_deploy_dir

    - name: Unarchive tarbal on webserver
      unarchive: src={{ app_local_path }} dest={{ deploy_dir }}
      when: not new_deploy_dir.stat.exists

    - name: Set file permissions
      file: path={{ deploy_dir }} owner={{ deploy_user }} group={{ deploy_group }} recurse=yes mode=0700

    - name: Start app
      raw: sudo /bin/systemctl restart {{ app_name}}

From the project root, run:

export MIX_ENV=prod
mix ansible.playbook deploy

If all went according to plan, you should be able to navigate to your ec2 instance and see the app running. Navigate to http://ec2-xx-xx-xx-xx.us-west-1.compute.amazonaws.com/version and you'll see the current version of the app.

---

Deploying Updates

Let's bump the version of our app and see how to deploy changes:

# ./mix.exs
defmodule WebhookProcessor.MixProject do
  use Mix.Project

  def project do
    [
      app: :webhook_processor,
      version: "0.2.0",
      ...
    ]
  end
  ...
end

The we re-run our build/deploy workflow:

export MIX_ENV=prod
mix do docker.build prod
mix ansible.playbook deploy

Now navigate to http://ec2-xx-xx-xx-xx.us-west-1.compute.amazonaws.com/version and you will see your latest changes deployed.

---

To recap we've:

1. Installed and configured Ansible.
2. Created Ansible playbooks to setup a new server and deploy.
3. Created a mix task to facilitate everything.

With this approach you will now be able to standup the entire infrastructure needed to run your app. As usual, the full code can be found on github: https://github.com/jonlunsford/webhook_processor

Up next, let's look at another method for deploying elixir apps, we'll walk through getting setup with dokku on DigitalOcean and deploy a Phoenix app this time.

Comments

[Michael]

You rock! Thanks for the introduction to ansible deployment.

Would it be much more work to deploy Traefik in front of the elixir app?