DEV Community: Jon Lunsford

Refactoring Ruby: Introduce Null Object

Jon Lunsford — Fri, 01 Mar 2024 21:56:19 +0000

Introduce Null Object is a great refactoring tool when you find yourself checking for nil too often. nil checks are often signs of much larger problems, as it violates Tell, Don’t Ask, and results in defensive, bug prone programming. Let’s take a look at a simple example of refactoring nil checks:

class Movie
  attr_accessor :account

  def initialize(price)
    @price = price
  end

  def rent
    unless account.nil?
      account.charge(@price)
    end
  end

  def rentable?
    account && account.rentable?
  end

  def late_fees
    account.try(:late_fees) || 0
  end
end

This Movie class is incredibly defensive, it goes to great lengths to ensure its account actually exists. Why should that responsibility fall on Movie? It would be much better for Movie to just assume it has an account OR something resembling an account. Let’s Introduce Null Object:

class Movie
  attr_accessor :account

  def initialize(price)
    @price = price
  end

  def rent
    account.charge(@price)
  end

  def rentable?
    account.rentable?
  end

  def late_fees
    account.late_fees
  end

  def account
    # Introduce Null Object
    @account ||= NullAccount.new
  end
end

# Object that resembles an 'Account'
class NullAccount
  def charge(_price)
    "No Charge"
  end

  def rantable?
    false
  end

  def late_fees
    0
  end
end

We have gotten rid of all conditions defending against nil and introduced one that always returns an Account like object:

def account
  @account ||= NullAccount.new
end

Let’s run through the exact steps to take when applying Introduce Null Object:

Identify conditions that check for nil, that are defending against objects potentially not existing, for example:

def rent
  # nil check
  unless account.nil?
    account.charge(@price)
  end
end

Create an object that acts like the one you are checking for as nil. Specifically, create a new class and define all of the methods that are expected to be there, for example:

class NullAccount
  def charge(_price)
    "No Charge" 
  end
end

Finally, substitute the null object when the expected one does not exist, for example:

Class Movie
  ...
  def account
    @account ||= NullAccount.new
  end
end

Null objects are great at replacing conditional logic and making code more readable, as you don’t have to think through all of the different branches conditions introduce.

Refactoring Ruby: Replace Temp with Query

Jon Lunsford — Thu, 26 Oct 2023 17:45:38 +0000

Let’s look at another simple yet powerful refactoring method, Replace Temp with Query. You can use this when you have a temporary variable holding the result of an expression. I will reach for this when temporary variables add too much to a method’s size/complexity, or when the result of an expression needs to be used in other methods. It's a fantastic next step after the last approach we look at: Extract Variable

Let’s use Replace Temp with Query in the Movie#total_price method.

Before:

class Movie
  TAX_PERCENTAGE = 0.5

  def initialize(price:, discount_percentage: 0)
    @price = price
    @discount_percentage = discount_percentage
  end

  def total_price
    # Temp variables holding expressions
    tax = price * TAX_PERCENTAGE
    discount = price * (discount_percentage / 100.0)

    price + tax - discount
  end

  private

  attr_reader :price, :discount_percentage
end

After:

class Movie
  TAX_PERCENTAGE = 0.5

  def initialize(price:, discount_percentage: 0)
    @price = price
    @discount_percentage = discount_percentage
  end

  def total_price
    # Temp variables replaced with query methods
    price + tax - discount
  end

  private

  attr_reader :price, :discount_percentage

  def tax
    price * TAX_PERCENTAGE
  end

  def discount
    price * (discount_percentage / 100.0)
  end

The method before refactoring looks like:

def total_price
  # Temp variables holding expressions
  tax = price * TAX_PERCENTAGE
  discount = price * (discount_percentage / 100.0)
  price + tax - discount
end

The method after refactoring looks like:

def total_price
  # Temp variables replaced with query methods
  price + tax - discount
end

With the temp variables extracted to private query methods, there’s even less to grok when understanding the role of total_price.

Let’s run through the exact steps to take when applying Replace Temp with Query:

Extract the expression into a private method. Try to use a private method as we don’t want to change the public API of an object if we don’t have to.
Replace all references to the temp with the method name. Additionally, use the new query methods in any other places that need them.

Reach for this when you need to share the result of an expression or to reduce the size/complexity of a method.

Refactoring Ruby: Extract Method

Jon Lunsford — Tue, 10 Oct 2023 15:31:45 +0000

Last time, we looked at possibly one of the smallest refactorings you can do, Extract Variable. Let's take another tiny step and look at the Extract Method pattern.

Extract Method can be used when you have several expressions, a code fragment, that can be grouped together. Even if there is one expression that could be better explained with a well thought out method name, it’s still a good move.

Let’s use Extract method on the Invoice#print method:

Before:

class Invoice
  ...

  def print
    print_branding

    # print line items
    line_items.each do |line_item|
      puts "#{line_item.name}: #{line_item.price}"
    end

    # print subtotal
    puts "Subtotal: #{line_items.sum(&:price)}"
  end
end

After:

class Invoice
  ...

  def print
    print_branding
    print_line_items
    print_subtotal
  end

  private

  def print_line_items
    line_items.each do |line_item|
      puts "#{line_item.name}: #{line_item.price}"
    end
  end

  def print_subtotal
    puts "Subtotal: #{line_items.sum(&:price)}"
  end
end

The method before refactoring looks like:

def print
  print_branding
  # print line items
  line_items.each do |line_item|
    puts "#{line_item.name}: #{line_item.price}"
  end
  # print subtotal
  puts "Subtotal: #{line_items.sum(&:price)}"
end

Often times, if you see comments explaining what a code fragment does, it’s a smell indicating there is a method to be extracted.

The method after refactoring looks like:

def print
  print_branding
  print_line_items
  print_subtotal
end

Let’s walk through the steps used when applying Extract Method:

Identify candidate fragments by looking for some common smells like, expressions that cluster together, comments explaining the intent, or if a good method name could add clarity.
Turn those fragments into a method whose name explains the purpose of the method.

Reach for this when a method contains groups of seemingly un-related code fragments, or when better naming will make it easier for you or your coworkers to understand the intent of code.

Refactoring Ruby: Extract Variable

Jon Lunsford — Tue, 12 Sep 2023 01:49:07 +0000

Let’s take a look at a simple way to add clarity to your Ruby methods: Extract Variable or Introduce Explaining Variable. It's probably one of the least invasive refactoring methods. However, don’t let the simplicity fool you; any clarity or explicitness gained is well worth the change.

When you have complex expressions grouped together in a method, it's best to describe them as clearly and verbosely as necessary using the Extract Variable technique. Let's apply this approach to the Movie#total_price method in the code example below:

class Movie
  TAX_PERCENTAGE = 0.5

  def initialize(price:, discount_percentage: 0)
    @price = price
    @discount_percentage = discount_percentage
  end

  def total_price
    # BAD
    price + (price * TAX_PERCENTAGE) - (price * (discount_percentage / 100.0))
  end

  private

  attr_reader :price, :discount_percentage
end

class Movie
  TAX_PERCENTAGE = 0.5

  def initialize(price:, discount_percentage: 0)
    @price = price
    @discount_percentage = discount_percentage
  end

  def total_price
    # GOOD
    tax = price * TAX_PERCENTAGE
    discount = price * (discount_percentage / 100.0)

    price + tax - discount
  end

  private

  attr_reader :price, :discount_percentage
end

The bad version looks like:

def total_price
  # BAD
  price + (price * TAX_PERCENTAGE) - (price * (discount_percentage / 100.0))
end

While you can deduce that this method returns the total price of a movie, it takes too much effort to understand how it does so.

To improve this method, we'll apply the Extract Variable pattern:

def total_price
  # GOOD
  tax = price * TAX_PERCENTAGE
  discount = price * (discount_percentage / 100.0)

  price + tax - discount
end

In the improved "good" version, we introduce two explaining variables: tax and discount.

Now, at a glance we know the total price of a movie consists of price + tax — discount, and we now have clear, concise descriptions of those complex expressions.

By applying the Extract Variable refactoring pattern, we've made the code more readable and understandable, enhancing its overall clarity.

Deploying Elixir (3 of 3): Provisioning EC2 With Ansible

Jon Lunsford — Thu, 18 Jun 2020 18:00:23 +0000

This post is the third in a series of three on deploying elixir:

In pt 2. we installed Terraform and built some infrastructure on AWS, specifically a Debian Buster EC2 instance to run our app. Now that we have our release built, and a place to run our app, let's walk through the basics of Ansible and provision our new EC2 instance.

If you did not follow along with that last post, you can grab the complete code here: https://github.com/jonlunsford/webhook_processor

Ansible is:

A radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.

Installing Ansible

If you have pip installed, you can run:

$ pip install --user ansible

Otherwise take a look at their Installation Guide for in depth instructions on how to install on your system. Verify the install worked properly by opening a new shell and typing:

$ ansible --version

You should see some output about the install, mine shows:

ansible 2.8.0
...

Here's the outline of the tasks we will automate with Ansible:

Setup: Bootstrapping the new server with all the prerequisites
- Ensure python is installed (Ansible is a python lib)
- Create our deploy user/group
- Create our directory structure
- Upload various system files (sudoers, systemd)
- Forward traffic from port 80 to our running app
Deploy: Upload our build artifacts
- Unzip our app tarball in the correct directory
- Start/restart the app
Profit!

Finally, we'll wrap this all up in mix task so we can run:

mix ansible.playbook setup
mix ansible.playbook deploy

Configuring Ansible

Create an inventory file noting the public dns of your ec2 instance, this tells Ansible the hosts we would like to run commands on:

# ./rel/ansible/inventory/main.yml

---
all:
  children:
    webservers:
      hosts:
        ec2-x-xxx-x-xx.us-west-1.compute.amazonaws.com

Next, we'll use an ansible.cfg file to store our configuration:

# ./rel/ansible/ansible.cfg

[defaults]
inventory=./inventory/main.yml
remote_user=admin 
private_key_file=../webhook_processor_key

With these few files in place, we can now test to see if Ansible can connect to our ec2 instance, cd into ./rel/ansible and run:

ansible webservers -m ping

Notice we explicitly called the command with webservers, this matches our inventory file above and scopes our command to only run on those hosts. You could have further inventory like dbservers (or whatever is relevant) and you would scope any commands or playbook runs accordingly.

The last configuration step we will do is create a shared vars file under ./rel/ansible/vars/main.yml. This way we can populate and use common vars across our tasks:

# ./rel/ansible/vars/main.yml

---
mix_env: "{{ lookup('env', 'MIX_ENV')}}"
app_port: "{{ lookup('env', 'APP_PORT') }}"
app_name: "{{ lookup('env', 'APP_NAME') }}"
app_vsn: "{{ lookup('env', 'APP_VSN') }}"
app_local_release_path: "{{ lookup('env', 'APP_LOCAL_RELEASE_PATH') }}"

app_local_path: "{{ app_local_release_path }}/{{ mix_env }}-{{ app_vsn }}.tar.gz"

# Main app server config
# OS user that deploys / owns the release files
deploy_user: deploy
# OS group that deploys / owns the release files
deploy_group: "{{ deploy_user }}"

# Base directory for deploy files
deploy_dir: "/opt/{{ app_name }}"

# Dirs owned by the deploy user
deploy_dirs:
  - "{{ deploy_dir }}"

Notice we'll use the lookup('env', 'MY_ENV_VAR') function to grab a few details about our app, more on that when we write the mix task later.

Setup Tasks

Now that we've verified we can connect, we can begin provisioning our ec2 instance. First, we'll write a playbook that runs various setup tasks:

# ./rel/ansible/tasks/setup.yml

---
- hosts: webservers
  gather_facts: False
  become: yes
  vars_files:
    - "../vars/main.yml"
  tasks:
    # Ensure python is installed, the debian ec2 instance does by default
    # Ansible will show us a warning about that
    - name: Install python 2
      raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)

    # Add the deploy group set in ../vars/main.yml if it does not exists
    - name: "Add {{ deploy_group }} group"
      group: name={{ deploy_group }} state=present

    # Add the deploy user set in ../vars/main.yml if it does not exists
    - name: "Add {{ deploy_user }} user"
      user: name={{ deploy_user }} groups={{ deploy_group }} append=yes state=present

    # Create the directories for the app
    - name: Create deploy dirs
      file: path={{ item }} state=directory owner={{ deploy_user }} group={{ deploy_user }} mode=0700
      with_items: "{{ deploy_dirs }}"

    # Create a sudoers file that gives the app user ONLY the ability to
    # start/stop the app. This helps ensure the security of the server, in case
    # the app user were compromised
    - name: Create sudoers config for deploy user
      template:
        src: "../templates/sudoers.j2"
        dest: /etc/sudoers.d/{{ deploy_user }}-{{ app_name }}
        owner: root
        group: root
        mode: 0600

    # Upload our systemd template
    - name: Copy systemd config file
      template:
        src: "../templates/systemd.j2"
        dest: "/etc/systemd/system/{{ app_name }}.service"
        owner: root
        group: root
        mode: 0644

    # Enable the systemd service
    - name: Enable service
      service: name={{ app_name }} enabled=yes

    # Forward port 80 to our app_port, set in ../vars/main.yml
    - name: "Forward port 80 to {{ app_port }}"
      iptables:
        table: nat
        chain: PREROUTING
        in_interface: eth0
        protocol: tcp
        match: tcp
        destination_port: "80"
        jump: REDIRECT
        to_ports: "{{ app_port }}"
        comment: "Redirect web traffic to port {{ app_port }}"

We're using a couple template files here as well, let's create those next, first the systemd service file that will allow the app to be managed by systemctl:

## ./rel/ansible/templates/systemd.j2

[Unit]
Description={{ app_name }}
After=local-fs.target network.target

[Service]
Type=simple
User={{ deploy_user }}
Group={{ deploy_group }}
WorkingDirectory={{ deploy_dir }}
ExecStart={{ deploy_dir }}/bin/{{ mix_env }} start
ExecStop={{ deploy_dir }}/bin/{{ mix_env }} stop
EnvironmentFile={{ deploy_dir }}/{{ app_name }}.env
LimitNOFILE=65536
UMask=0027
SyslogIdentifier={{ deploy_user }}
Restart=always
RestartSec=5
RemainAfterExit=no

[Install]
WantedBy=multi-user.target

Next, the sudoers file that will only permit the app user (deploy) to stop/start/restart the app:

## ./rel/ansible/templates/sudoers.j2

# {{ ansible_managed }}
{{ deploy_user }} ALL=(ALL) NOPASSWD: /bin/systemctl start {{ app_name }}, /bin/systemctl stop {{ app_name }}, /bin/systemctl restart {{ app_name }}, /bin/systemctl status {{ app_name }}, /bin/systemctl status {{ app_name }}
Defaults:{{ deploy_user }} !requiretty

The last template we'll need is our env file, allowing us to set any vars the app may need. In this case, just the app_port:

## ./rel/ansible/templates/env.j2

APP_PORT={{ app_port }}

Ansible Mix Task

Since running these commands will become redundant, let's write yet another Mix task to facilitate everything, including setting all the env vars ./rel/ansible/vars/main.yml is picking up. Create the initial namespace:

# ./lib/mix/tasks/ansible.playbook.ex

defmodule Mix.Tasks.Ansible.Playbook do
  use Mix.Task

  @shortdoc "Run ansible playbooks"
  def run(args) do
    Mix.Task.run("ansible", args)
  end
end

Our task will need to setup a few env vars, then kick off the ansible playbook:

# ./rel/ansible/tasks/ansible.ex

defmodule Mix.Tasks.Ansible do
  use Mix.Task
  use Mix.Tasks.Utils

  @shortdoc "Run ansible playbooks"
  def run([playbook]) do
    cmd_args = ["./rel/ansible/tasks/#{playbook}.yml"]
    {dir, _resp} = System.cmd("pwd", [])
    dir = String.trim(dir)
    mix_env = System.get_env("MIX_ENV")

    System.cmd(
      "ansible-playbook",
      cmd_args,
      env: [
        {"ANSIBLE_CONFIG", "#{dir}/rel/ansible/ansible.cfg"},
        {"APP_NAME", app_name()},
        {"APP_VSN", app_vsn()},
        {"APP_PORT", app_port()},
        {"MIX_ENV", mix_env},
        {"APP_LOCAL_RELEASE_PATH", "#{dir}/_build/#{mix_env}"}
      ],
      into: IO.stream(:stdio, :line)
    )
  end
end

Notice we're setting all the env vars that are relevant to the app, that will become available to Ansible. Run the setup command now, from the project root:

export MIX_ENV=prod
mix ansible.playbook setup

You should see all of the output indicating the result of each task.

Deploying the App Tarball

Once setup is complete, the next step is a matter of uploading our build artifacts, here's the entire playbook needed to actually deploy:

# ./rel/ansible/tasks/deploy.yml

---
- hosts: webservers
  become: yes
  vars_files:
    - "../vars/main.yml"
  tasks:
    - name: Copy env to webserver
      template:
        src: "../templates/env.j2"
        dest: "{{ deploy_dir }}/{{ app_name }}.env"

    - name: "Check if {{ deploy_dir }} exists"
      stat:
        path: "{{ deploy_dir }}/{{ app_name }}.tar.gz"
      register: new_deploy_dir

    - name: Unarchive tarbal on webserver
      unarchive: src={{ app_local_path }} dest={{ deploy_dir }}
      when: not new_deploy_dir.stat.exists

    - name: Set file permissions
      file: path={{ deploy_dir }} owner={{ deploy_user }} group={{ deploy_group }} recurse=yes mode=0700

    - name: Start app
      raw: sudo /bin/systemctl restart {{ app_name}}

From the project root, run:

export MIX_ENV=prod
mix ansible.playbook deploy

If all went according to plan, you should be able to navigate to your ec2 instance and see the app running. Navigate to http://ec2-xx-xx-xx-xx.us-west-1.compute.amazonaws.com/version and you'll see the current version of the app.

Deploying Updates

Let's bump the version of our app and see how to deploy changes:

# ./mix.exs
defmodule WebhookProcessor.MixProject do
  use Mix.Project

  def project do
    [
      app: :webhook_processor,
      version: "0.2.0",
      ...
    ]
  end
  ...
end

The we re-run our build/deploy workflow:

export MIX_ENV=prod
mix do docker.build prod
mix ansible.playbook deploy

Now navigate to http://ec2-xx-xx-xx-xx.us-west-1.compute.amazonaws.com/version and you will see your latest changes deployed.

To recap we've:

Installed and configured Ansible.
Created Ansible playbooks to setup a new server and deploy.
Created a mix task to facilitate everything.

With this approach you will now be able to standup the entire infrastructure needed to run your app. As usual, the full code can be found on github: https://github.com/jonlunsford/webhook_processor

Up next, let's look at another method for deploying elixir apps, we'll walk through getting setup with dokku on DigitalOcean and deploy a Phoenix app this time.

Deploying Elixir (2 of 3): Terraforming an AWS EC2 Instance

Jon Lunsford — Thu, 18 Jun 2020 17:59:03 +0000

This post is the second in a series of three on deploying elixir:

In Pt. 1 we built our production release, locally, with Docker. Now that we have a release, we need somewhere to run it. Let's walk through the basics of Terraform, by HashiCorp. By using Terraform, we can write our infrastructure as code, albeit a very simple infrastructure for our webhook processor app.

If you did not follow along with that last post, you can grab the complete code here: https://github.com/jonlunsford/webhook_processor

Terraform is:

A workflow to provision infrastructure for private cloud, public cloud, and external services. Build reusable Terraform templates to define the topology of infrastructure using code.

The goals for this post are:

Install Terraform
Configure Terraform for AWS
Create an EC2 instance

Installing Terraform

If you're using homebrew on OSX, you can run:

$ brew install terraform

Otherwise, you can grab a binary for your system here. Verify the install worked properly by opening a new shell and typing:

$ terraform

You should see some usage output. If you see an error, it's possible the binary is not in your $PATH, ensure that the directory Terraform was installed in, is part of your $PATH.

Configuring Terraform for AWS

With Terraform installed, we can begin building infrastructure on AWS. You'll need an AWS account, grab a free one here if you don't already have one. Next you'll need valid credentials, which can be found in the IAM Management Console.

Finally, the last prerequisite is having an ssh key pair ready to supply, so we can eventually connect to our instance, both manually and through Ansible, outlined in the next post. Let's generate a new key to be used with our instance:

ssh-keygen -f ./rel/webhook_processor_key

Configuration is composed in *.tf files, this is how we will describe the infrastructure we want. The following is the entire config file to build an EC2 instance with a new key pair and security group allowing http+ssh access, save this file at ./rel/terraform/webhook_processor.tf:

# ./rel/terraform/webhook_processor.tf

provider "aws" {
  region = "us-west-1" # Set the region available to you
  access_key = ACCESS_KEY_HERE
  secret_key = SECRET_KEY_HERE
}

# A security group is required to configure allowed traffic on the instance
resource "aws_security_group" "webhook_processor_sg" {
  name          = "allow-all-sg"
  description   = "Allow all inbound ssh+http traffic"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = [ "0.0.0.0/0" ]
  }

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = [ "0.0.0.0/0" ]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# A key pair is required to ultimately allow SSH access
resource "aws_key_pair" "webhook_processor_kp" {
  key_name   = "webhook_processor_deploy"
  public_key = file("../webhook_processor_key.pub") # Read from the file we generated
}

# Finally, create the ec2 instance using the above references
resource "aws_instance" "webhook_processor" {
  ami             = "ami-0e4ae7403dc481431" # Debian Buster AMI
  instance_type   = "t2.micro" # Should be eligible for free
  key_name        = aws_key_pair.webhook_processor_kp.key_name
  security_groups = [aws_security_group.webhook_processor_sg.name]
}

output "webhook_processor_host" {
  value = aws_instance.webhook_processor.public_dns
}

Replace ACCESS_KEY_HERE and SECRET_KEY_HERE with your own. You can hard code them this way for now or place your credentials in ~/.aws/credentials, that file should look like:

[default]
aws_access_key_id = ACCESS_KEY_HERE
aws_secret_access_key = SECRET_KEY_HERE

If the access_key and secret_key are omitted from the provider block, Terraform will automatically look for the above file, in which case it will look like:

provider "aws" {
  region = "us-west-1"
}

NOTE: If you use a region other than us-west-1 the AMI for Debian Buster will be different, AMI's are region specific. There are a few ways to find an AMI on AWS. Once you've found an AMI, be sure to replace the example above.

Let's go over the .tf file really quick. provider blocks are used to configure named providers, aws in our case. There are a whole bunch of providers available. Terraform files can have many providers configured as well, you can begin to see how you can describe very complex infrastructure topologies.

The resource block is used to describe a resource that our architecture requires, in this case we will need an aws_security_group, aws_key_pair and an aws_instance.

Finally, output blocks allow you to retrieve specific attributes of your resources. We will need to know the public_dns (url) of our instance when we deploy our app.

Initializing Terraform

Now we are ready to confirm our .tf file is correct. cd into ./rel/terraform/ and run the following:

$ terraform init

This will prepare Terraform, including downloading any provider plugins it needs, you should see it downloading the provider.aws plugin.

Creating Infrastructure

After successful initialization we can now create our infrastructure, from the same directory (./rel/terraform/) run:

$ terraform apply

You will see an overview of the execution plan along with a prompt to confirm applying the plan, something like:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

 ...

Notice + create, this indicates the type of action Terraform will perform. Since our infrastructure does not exist, it assumes we are creating new resources.

Changing Infrastructure

To get a little more comfortable with Terraform, let's change something and see how that is handled. Update the ami for our aws_instance:

resource "aws_instance" "webhook_processor" {
  ami             = "ami-29918949" # Different resource
  instance_type   = "t2.micro" # Should be eligible for free
  key_name        = aws_key_pair.webhook_processor_kp.key_name
  security_groups = [aws_security_group.webhook_processor_sg.name]
}

Again, run:

$ terraform apply

And again, we see an overview of the execution plan with a prompt to confirm:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

...

Notice -/+ destroy and then create replacement, changing the actual instance ami requires destroying the existing one and creating a new one.

Enter yes at the prompt to execute the plan.

Destroying Infrastructure

Lastly, let's see how Terraform can destroy resources it manages, run the following:

$ terraform destroy

Here's the overview of that command:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

...

We've now gone through the general workflow with Terraform. Go ahead and recreate an EC2 instance, so we have something to work with in part 3. Finally, let's confirm our ssh access is permitted, after you recreate the ec2 instance, test it out using the webhook_processor_host value that was printed out after, mine looks like:

ssh admin@ec2-xx-xxx-x-xxx.us-west-1.compute.amazonaws.com -i ../webhook_processor_key "cat ~/.ssh/authorized_keys"

You should see the webhook_processor_deploy key printed out.

Conclusion

We've seen how to describe our infrastructure topology with Terraform and how to apply that to our providers, AWS in this case. You could just as easily apply this to other providers, DigitalOcean for example.

Up next, we will provision our newly created EC2 instance with Ansible and deploy our app.

As always, the code is available on GitHub: https://github.com/jonlunsford/webhook_processor

Deploying Elixir (1 of 3): Building Releases With Mix

Jon Lunsford — Thu, 18 Jun 2020 17:58:45 +0000

This post is the first in a series of three on deploying elixir:

Last time, we built a small JSON endpoint with Plug, Cowboy and Poison, let's retrofit that same app to build releases with Docker & Mix. The first step in deploying Elixir apps is to build them on the same target they will run. There are many ways to achieve this, we will walk through using Docker, locally, to build releases that can then be deployed.

If you did not follow along with that last post, you can grab the complete code here: https://github.com/jonlunsford/webhook_processor

Mix is:

A build tool that provides tasks for creating, compiling, and testing Elixir projects, managing its dependencies, and more.

Mix >= 1.9 introduced a task to manage releases, providing a built in alternative to tools like Distillery. We will be using this task to manage our releases: Mix.Tasks.Release

This is how we will package our app for release. We will also write a Mix task to:

Build a Docker build image.
Build releases with Mix.

Preparing our App for Release

Before we build our release with Docker, let's make a few changes to our app to get it ready for building releases. Update the project function in mix.exs for release:

# ./mix.exs
defmodule WebhookProcessor.MixProject do
 ...
  def project do
    [
      app: :webhook_processor,
      version: "0.1.0",
      elixir: "~> 1.10",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      releases: [ # add releases configuration
        prod: [ # we can name releases anything, this will be prod's config
          include_executables_for: [:unix], # we'll be deploying to Linux only
          steps: [:assemble, :tar] # have Mix automatically create a tarball after assembly
        ]
      ]
    ]
  end
 ...
end

Now we can begin building our release for our target environment.

Add a /version Route

Let's make one more change to our endpoint so we can easily see which version of our app is ultimately running, this will come into play once we've actually deployed, add this to ./lib/webhook_processor/endpoint.ex:

defmodule WebhookProcessor.Endpoint do
  ...
  get "/version" do
    resp =
      case :application.get_key(:webhook_processor, :vsn) do
        {:ok, vsn} -> vsn
        _ -> "version not found :("
      end

    send_resp(conn, 200, resp)
  end
  ...
end

Building the Docker Image

Given that Erlang, and thus Elixir, are compiled, we must target the environment our app will run in at build time. Fortunately, Docker makes this trivial. The general idea is this:

Create a Dockerfile that matches your target environment.
Build the release with Mix inside the Docker container.
Deploy the produced artifacts, the app tarball.

So let's do that, we will be targeting a Debian environment. The minimal Dockerfile will look like this:

# ./Dockerfile
# ENV matching production target host
# We'll be using a Debian linux EC2 instance to run this app
# See all official elixir docker images here: https://hub.docker.com/_/elixir
FROM elixir:1.10

# By default, if we're cutting a release it'll likely be prod
ARG ENV=prod

# We'll pass in ENV as a build arg to docker
ENV MIX_ENV=$ENV

# Our working directory within the container
WORKDIR /opt/build

# Add our release script to the container, named `release` and
# placed into the ./bin/ directory in our project root
ADD ./bin/release ./bin/release

# This is our entry point, make sure to run
# `chmod +x bin/release` to make this script executable
CMD ["bin/release", $ENV]

With this, we have what we need to build the Elixir image, next let's write our release bash script that will facilitate the Elixir build inside our image. Be sure to make this file executable by running chmod +x ./bin/release:

#!/usr/bin/env bash

set -e

# The one arg that we can accept will be the ENV we are building
export MIX_ENV=$1

echo "Starting release process..."
cd /opt/build

echo "Installing rebar and hex..."
mix local.rebar --force
mix local.hex --if-missing --force

echo "Fetching project deps..."
mix deps.get --only prod

echo "Cleaning any leftover artifacts..."
mix do clean, compile --force

echo "Building $1 release..."
mix release $1 --overwrite

echo "Build completed!"
exit 0

Building the Release

Given that building releases will be so common, let's write a Mix task to handle the redundancy. Our goal will be to have a task that reads like:

$ mix docker.build prod

It's common to create namespaces for mix tasks by defining them in their own file, here's how we get the docker.build task:

# ./lib/mix/tasks/docker.build.ex
defmodule Mix.Tasks.Docker.Build do
  use Mix.Task

  @shortdoc "Docker utilities for building releases"
  def run(args) do
    Mix.Task.run("docker", args)
  end
end

The run function simply runs the docker task, passing along the args. The upper Docker module contains the build code:

# ./lib/mix/tasks/docker.ex
defmodule Mix.Tasks.Docker do
  use Mix.Task
  use Mix.Tasks.Utils

  @shortdoc "Docker utilities for building releases"
  def run([env]) do
    # Build a fresh Elixir image, in case Dockerfile has changed
    build_image(env)

    # Get the current working directory
    {dir, _resp} = System.cmd("pwd", [])

    # Mount the working directory at /opt/build within the new elixir image
    # Execute the /opt/build/bin/release script
    docker(
      "run -v #{String.trim(dir)}:/opt/build --rm -i #{app_name()}:latest /opt/build/bin/release #{env}"
    )
  end

  defp build_image(env) do
    docker("build --build-arg ENV=#{env} -t #{app_name()}:latest .")
  end

  defp docker(cmd) do
    System.cmd("docker", String.split(cmd, " "), into: IO.stream(:stdio, :line))
  end
end

Here, we have a run function that matches the pattern {env}, you can now build your application by running:

$ mix docker.build prod

After the script runs, you will have a tarball containing everything it needs to run on our debian target at:

_build/prod/prod-0.1.0.tar.gz

To recap, we've:

Created a Dockerfile to match our Debian target environment.
Created a build script that will run mix.
Created a mix task to facilitate everything.

With this approach you should now be able to build releases to match your target environment. As usual, the full code can be found on github: https://github.com/jonlunsford/webhook_processor

Up next, Terraforming an EC2 instance to deploy our app to.

Elixir: Building a Small JSON Endpoint With Plug, Cowboy and Poison

Jon Lunsford — Fri, 10 May 2019 15:58:48 +0000

There’s been many times when I’ve just wanted to add a simple JSON endpoint to an app to, expose a service, or process webhook events, without the overhead of a full framework. Let’s see how easy it is to build a production ready endpoint with Plug, using Erlang’s Cowboy HTTP server.

Plug Is:

A specification for composable modules between web applications

Connection adapters for different web servers in the Erlang VM

If you’re coming from Ruby/Rails, think Rack, from Node, think Express, et al. Of course the concepts of these libraries are similar on the surface, they are unique in their own rights.

Cowboy Is:

A small, fast and modern HTTP server for Erlang/OTP

Additionally, it’s a fault tolerant “server for the modern web” supporting HTTP/2, providing a suite of handlers for Websockets and interfaces for long-lived connections. Without going into too much more detail, it’s safe to say, it’s an acceptable choice for production. Consult the docs for more info.

Poison Is:

A JSON library for Elixir focusing on wicked-fast speed without sacrificing simplicity, completeness, or correctness.

In other words, it’s a super fast, reliable JSON parsing library.

Building the Endpoint

With the short definitions out of the way, let’s build an endpoint to process incoming webhook events. Now, we want this to be “production ready”, what does that mean for our use case?

Fault tolerant: Always available. Can never go down (at least not easily :)
Easily configurable: Can be deployed to any environment
Well tested: Give us confidence in what we’re shipping

We do have a very simple use case for this, it’s a good idea to understand your own requirements before selecting tools and investing time into a solution.

1. Create a new, supervised, Elixir app:

$ mix new webhook_processor --sup
$ cd webhook_processor

--sup will create an app suitable for use as an OTP application. OTP and supervision will give us our #1 requirement from above. Our server will be supervised and restarted automatically in the event of a crash, while the server may crash the Erlang VM should not (at least not easily :).

2. Add Plug, Cowboy, and Poison as dependencies

# ./mix.exs
defmodule WebhookProcessor.MixProject do
  use Mix.Project

  def project do
    [
      app: :webhook_processor,
      version: "0.1.0",
      elixir: "~> 1.7",
      start_permanent: Mix.env() == :prod,
      deps: deps()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      # Add :plug_cowboy to extra_applications
      extra_applications: [:logger, :plug_cowboy],
      mod: {WebhookProcessor.Application, []}
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    [
      {:plug_cowboy, "~> 2.0"}, # This will pull in Plug AND Cowboy
      {:poison, "~> 3.1"} # Latest version as of this writing
    ]
  end
end

Couple notes here, we added plug_cowboy (in deps) as a single dependency for Plug AND Cowboy. We need to add :plug_cowboy to the extra_applications list (in application) as well.

3. Mix deps.get

$ mix deps.get

4. Implement application.ex

# ./lib/webhook_processor/application.ex
defmodule WebhookProcessor.Application do
  @moduledoc "OTP Application specification for WebhookProcessor"

  use Application

  def start(_type, _args) do
    # List all child processes to be supervised
    children = [
      # Use Plug.Cowboy.child_spec/3 to register our endpoint as a plug
      Plug.Cowboy.child_spec(
        scheme: :http,
        plug: WebhookProcessor.Endpoint,
        options: [port: 4001]
      )
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: WebhookProcessor.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

5. Implement WebhookProcessor.Endpoint

# ./lib/webhook_processor/endpoint.ex
defmodule WebhookProcessor.Endpoint do
  @moduledoc """
  A Plug responsible for logging request info, parsing request body's as JSON,
  matching routes, and dispatching responses.
  """

  use Plug.Router

  # This module is a Plug, that also implements it's own plug pipeline, below:

  # Using Plug.Logger for logging request information
  plug(Plug.Logger)
  # responsible for matching routes
  plug(:match)
  # Using Poison for JSON decoding
  # Note, order of plugs is important, by placing this _after_ the 'match' plug,
  # we will only parse the request AFTER there is a route match.
  plug(Plug.Parsers, parsers: [:json], json_decoder: Poison)
  # responsible for dispatching responses
  plug(:dispatch)

  # A simple route to test that the server is up
  # Note, all routes must return a connection as per the Plug spec.
  get "/ping" do
    send_resp(conn, 200, "pong!")
  end

  # Handle incoming events, if the payload is the right shape, process the
  # events, otherwise return an error.
  post "/events" do
    {status, body} =
      case conn.body_params do
        %{"events" => events} -> {200, process_events(events)}
        _ -> {422, missing_events()}
      end

    send_resp(conn, status, body)
  end

  defp process_events(events) when is_list(events) do
    # Do some processing on a list of events
    Poison.encode!(%{response: "Received Events!"})
  end

  defp process_events(_) do
    # If we can't process anything, let them know :)
    Poison.encode!(%{response: "Please Send Some Events!"})
  end

  defp missing_events do
    Poison.encode!(%{error: "Expected Payload: { 'events': [...] }"})
  end

  # A catchall route, 'match' will match no matter the request method,
  # so a response is always returned, even if there is no route to match.
  match _ do
    send_resp(conn, 404, "oops... Nothing here :(")
  end
end

This looks like a lot, but most of this file is just some helpful comments. The gist is, we are using the macros, get and post, from Plug.Router to generate our routes. This module is a plug itself, and it defines its own plug pipeline. Note, match and dispatch are required in order for us to handle requests and dispatch responses. Pipeline is a key concept here, as the order of plugs determines the order of operations. Notice that match is before we define our parser, this means we will not parse anything unless there is a route match. IF the order was reversed, we’d be trying to parse requests regardless of routes matching. Refer to the docs on Plug.Router for more info.

6. Make the Endpoint Configurable

# ./lib/webhook_processor/application.ex
defmodule WebhookProcessor.Application do
  @moduledoc "OTP Application specification for WebhookProcessor"

  use Application

  def start(_type, _args) do
    # List all child processes to be supervised
    children = [
      # Use Plug.Cowboy.child_spec/3 to register our endpoint as a plug
      Plug.Cowboy.child_spec(
        scheme: :http,
        plug: WebhookProcessor.Endpoint,
        # Set the port per environment, see ./config/MIX_ENV.exs
        options: [port: Application.get_env(:webhook_processor, :port)]
      )
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: WebhookProcessor.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

We’ve swapped out the hard coded port value of the Cowboy options for an environment variable, this will allow us to run the webhook processor in any environment we need to. Finally, create a config file for each MIX_ENV you need:

#./config/config.exs

# This file is responsible for configuring your application
# and its dependencies with the aid of the Mix.Config module.
use Mix.Config

import_config "#{Mix.env()}.exs"

-------------------

# ./config/dev.exs

use Mix.Config

config :webhook_processor, port: 4001

-------------------

# ./config/test.exs

use Mix.Config

config :webhook_processor, port: 4002

-------------------

# ./config/prod.exs

use Mix.Config

config :webhook_processor, port: 80

7. Test

# ./test/webhook_processor/endpoint_test.exs
defmodule WebhookProcessor.EndpointTest do
  use ExUnit.Case, async: true
  use Plug.Test

  @opts WebhookProcessor.Endpoint.init([])

  test "it returns pong" do
    # Create a test connection
    conn = conn(:get, "/ping")

    # Invoke the plug
    conn = WebhookProcessor.Endpoint.call(conn, @opts)

    # Assert the response and status
    assert conn.state == :sent
    assert conn.status == 200
    assert conn.resp_body == "pong!"
  end

  test "it returns 200 with a valid payload" do
    # Create a test connection
    conn = conn(:post, "/events", %{events: [%{}]})

    # Invoke the plug
    conn = WebhookProcessor.Endpoint.call(conn, @opts)

    # Assert the response
    assert conn.status == 200
  end

  test "it returns 422 with an invalid payload" do
    # Create a test connection
    conn = conn(:post, "/events", %{})

    # Invoke the plug
    conn = WebhookProcessor.Endpoint.call(conn, @opts)

    # Assert the response
    assert conn.status == 422
  end

  test "it returns 404 when no route matches" do
    # Create a test connection
    conn = conn(:get, "/fail")

    # Invoke the plug
    conn = WebhookProcessor.Endpoint.call(conn, @opts)

    # Assert the response
    assert conn.status == 404
  end
end

These tests are pretty simple, but they confirm our server is working as expected. One could argue that the only thing we should care about from these tests are the response codes, not the side effects that happen when events are processed. Always test UP TO the boundaries of your module, never beyond, unless you are writing an integration style test.

Conclusion

With very little effort we’ve built a small but mighty endpoint. Thanks to Cowboy, you should be able to serve more connections from one server than you’ll likely ever need, so let’s add low cost to the list of benefits as well.

What about deployment? Let's walk through building releases and deploying to AWS:

As always, the code is available on GitHub: https://github.com/jonlunsford/webhook_processor