CircleCI + AWS How to create CI/CD pipeline from scratch Part 0 - Preparation

Introduction

In this series of posts, I'm going to show how to set up the CI/CD environment using AWS and CircleCI.
As a final result, you will get the pipeline that:

• runs tests after each commit;
• builds containers from development and master branches;
• pushes them into the ECR;
• redeploys development and production EC2 instances from the development and master containers respectively.

This guide consists of three parts:

• ➡️ Preparation - where I'll explain this workflow, and show how to prepare an application for it
• AWS - this chart is about how to set up the AWS environment from scratch;
• CircleCI - here I'll demonstrate how to automatize deployment process using circleci.com

Workflow

When you developing applications in "real life" you usually(but not always), sooner or later, found that you need to:

• 🤖 run automatic tests;
• 🔍 run different checks(code coverage, security audit, code style, etc.);
• 🧪 test how a feature works before you deploy it to the production;
• 💸 deliver results as fast as possible.

There is an example of a workflow that was used in many projects where I've participated. It works very well in small(3-8 person) teams:

1) Create a branch for a feature from master
2) Work
3) Push this feature
4) Optionally Run tests, checks, etc. for this branch
5) In case of success merge this branch to the development branch
6) Run everything again and redeploy the development server
7) Test it manually
8) Merge feature to the production branch
9) Redeploy production

Workflow example

I'm going to show how it can work on the workflow with two main branches:

• master - has tested, production-ready code(deploys on a production server after approving)
• development - based on the master and also has changes that being tested now(deploys on a development server right after changes on the github)

Prepare the application

As an example let's take a simple API server with one route written on the Elixir. I will not explain here how to create it, there is a fantastic post about. Or you can use my application, it is already configured and prepared.
There I'll focus only on the specific moments that are needed to prepare the server for work in this environment.

Prepare release application

This Elixir application I am going to deploy using mechanism of releases.
Briefly, it generates an artefact that contains the Erlang VM and its runtime, compiled source code and launch scripts.

---

Let me make a little digress to tell about the methodology I'm trying to follow for designing microservices.
I'm talking about 12 factor app manifest. It's a set of recommendations for building software-as-a-service apps that:

• Use declarative formats for setup automation, to minimize time and cost for new developers joining the project.
• Have a clean contract with the underlying operating system, offering maximum portability between execution environments.
• Are suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration.
• Minimize divergence between development and production, enabling continuous deployment for maximum agility.
• And can scale up without significant changes to tooling, architecture, or development practices.

---

One of these principles recommends us to store configurable parameters(ports, API keys, services addresses, etc.) in system environment variables. To configure our release application using them you should create the file config/releases.exs and describe these variables:

import Config

config :simple_plug_server, port: System.get_env("PORT")

More about different config files in Elixir applications you can find here
and here

Launch script

Next thing I would like to cover is starting an application. The most common way is to use a special shell script
for it that contains different preparation steps like waiting for a database, initializing system variables, etc. Also,
it makes your Docker file more expressive. I think you will agree that CMD ["bash", "./simple_plug_server/entrypoint.sh"] looks better than CMD ["bash", "cmd1", "arg1", "arg2", ";" "cmd2", "arg1", "arg2", "arg3"]. The entrypoint script for this server is very simple:

#!/bin/sh

bin/simple_plug_server start

This application works in the docker container so the last command bin/simple_plug_server start starts
app without daemonizing it and writes logs right into the stdout. That allows us to gather logs simpler(AWS Cloudwatch gathers these data without additional configs).

Dockerization

And on the last step, you should create the Dockerfile that builds result container. I prefer to use two steps builds for Elixir applications because result containers are very thin(approx. 50-70MB).

FROM elixir:1.10.0-alpine as build

# install build dependencies
RUN apk add --update git build-base

# prepare build dir
RUN mkdir /app
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV=prod

# install mix dependencies
COPY mix.exs mix.lock ./
COPY config config
RUN mix deps.get
RUN mix deps.compile

# build project
COPY lib lib
RUN mix compile

# build release
RUN mix release

# prepare release image
FROM alpine:3.12 AS app
RUN apk add --update bash openssl

RUN mkdir /app
WORKDIR /app

COPY --from=build /app/_build/prod/rel/simple_plug_server ./
COPY --from=build /app/lib/simple_plug_server/entrypoint.sh ./simple_plug_server/entrypoint.sh
RUN chown -R nobody: /app
USER nobody

ENV HOME=/app

CMD ["bash", "./simple_plug_server/entrypoint.sh"]

Finally you can build it using docker build ., and run docker run -it -p 4000:4000 -e PORT=4000 {IMAGE_ID}.
The server will be available on the localhost:4000 and will write logs to the stdout. 🎉

On the next part of this tutorial, I'll show how to automatically build images using CircleCI and push them to the Amazon ECR.

Comments

[Jack]

pog