Consider AWS Batch for Your Next Background Jobs

Christopher Lai — Sun, 30 Jun 2019 21:29:37 +0000

A colleague and I are currently in the process of updating an aging service in our stack. The service is pretty well isolated — it takes a bit of input, does a few units of work, then puts the results in a specific S3 bucket with a key provided as part of the input.

Normally I would reach for SQS + EC2 with a homegrown background worker or a community library like shoryuken. However, having a few dozen servers in production already, I wanted a solution that wouldn’t include additional infrastructure I would need to manage.

When I run in to an infrastructure related challenge nowadays, I do a quick search in the AWS console to see what turns up. Sure enough, AWS has a service for background jobs called Batch.

At first glance, it wasn't entirely clear how Batch worked. I Googled a few different keyword arrangements, none of which turned up anything particularly useful; a few articles here and there, but nothing like an in-depth tutorial that covered our particular use case.

After going through the Getting Started Guide, I found that Batch was very simple to set up. Most of the overhead was getting over what a traditional background worker setup looks like.

There are a few characteristics that, collectively, make Batch work well for background jobs.

Compute Environment

Batch jobs are run in a Compute Environment. This is just an ECS cluster that Batch creates and manages for you. These are the nodes/instances that will run your job.

Job Definition

Jobs are defined using a Job Definition, which allows you to customize the number of CPUs, memory, and retries, as well as set environment variables and other options, like what default Docker Image you want to use.

We knew we wanted to use Docker Images from the beginning. Our job has a large amount of runtime dependencies, and we thought it best to packages these into an atomic unit. Batch is a great fit, since Batch supports Docker Images.

Job Queue

Unlike other background job libraries, Batch has its own queueing system — it doesn't rely on SQS or Redis.

Job Queues are tied to a Compute Environment and can have different priorities. Jobs are submitted to Job Queues, which get run by the Compute Environment.

Job

Jobs are submitted to a Job Queue using a Job Definition. Default arguments set in the Job Definition can be overwritten when submitting a job.

There are a few ways to submit jobs. We submit our jobs in our codebase using the the Batch gem provided by AWS.

There's still a lot more for me to learn about Batch, but I wanted to share my experience using it so far. Feel free to leave a comment if you have any questions.

Deploying a Phoenix Project on AWS

Christopher Lai — Wed, 12 Jun 2019 04:34:40 +0000

This is a quick rundown of how we currently deploy a Phoenix project on AWS. It won’t include any code or screenshots — just a high-level overview.

While we use Distillery to build our release, I will not be covering those details here. I recommend reviewing the documentation for Distillery on Hex.

Overview

This is the basic flow of how a merge into master makes it to EC2.

GitHub > AWS CodePipeline > AWS CodeBuild > AWS CodeDeploy > AWS EC2

GitHub + AWS CodePipeline

AWS CodePipeline is triggered when a Pull Request is merged into master. This is done using GitHub Webhooks setup via CodePipeline. The setup is pretty simple and AWS has some great documentation on getting started.

CodePipeline then hands the code off to CodeBuild to build our release.

CodeBuild

We build our release using a custom Docker image on CodeBuild. The default Docker image provided by CodeBuild can be used, but we found that building a custom image was a better fit for our needs. Our custom image is very simple and consists of the following:

Ubuntu 18.04
NodeJS 10LTS
Erlang
Elixir

The release is then passed back to CodePipeline via the artifact configuration option for CodeBuild.

CodePipeline then triggers a deploy using CodeDeploy.

Note: Your Git history is not available when a build is triggered by CodePipeline.

CodeDeploy

CodeDeploy is used to push the release generated by CodeBuild in the previous step directly to our EC2 instances. This is done by a CodeDeploy agent running on each of our EC2 instances. The agent polls for available deployments and deploys them automatically if one is available.

All of our instances run a custom AMI that has the CodeDeploy agent pre-installed. We build our Custom AMIs using Packer and Ansible. Our EC2 instances are managed with an Auto Scaling Group using a custom Launch Configuration.

The entire process is simple and fast. It takes about 3-5 minutes to go from merged to fully deployed.

Bonus: Production credentials

Our goal was to treat our EC2 instances as cattle and not pets. This meant that we needed an automated and secure way of passing production credentials and secrets to our EC2 instances. We didn’t want to store a text file or have environment variables on each EC2 instance. We settled on AWS Secrets Manager instead.

I wrote a custom Distillery Config Provider that fetches credentials from Secrets Manager when the Phoenix release is started. Secrets Manager returns TOML to the custom Config Provider, then the TOML is then parsed, and the configurations are set using the Elixir Application module.

I would be happy to write up additional details or answer any questions.

DEV Community: Christopher Lai