On my first article we created a nice local development environment for Elixir and Phoenix with Docker and docker-compose. Now it's time to write some CI scripts and Terraform templates for our continuous integration and AWS infrastructure.
We are going to do things in the following order:
- Make sure you have all ready and set
- Create needed resources to AWS to push deploy-ready images to AWS (ECR, IAM users, policies)
- Write
.gitlab-ci.ymlexcept deploy stage - Create rest of the resources (ECS, LB, RDS etc.) in order to run our application
- Add deploy-stage to
.gitlab-ci.yml
Note: This is not production grade infrastructure setup, but rather one which can be used as a base.
Prerequisites
GitLab
- Phoenix application in GitLab repository
- Pipelines & Container Registry enabled in GitLab project settings
- Production ready Dockerfile for the Phoenix application/project
In case you don't have optimized Dockerfile for cloud deployments, read this.
AWS & Terraform
- AWS CLI configured with your credentials (~/.aws/credentials)
- Terraform CLI tool installed
Note: GitLab offers 400 minutes of CI time and container registry for free users (private and public repositories). (Updated 16.12. free CI minutes reduced from 2000 to 400)
Creating the initial resources
ECR stands for Elastic Container Registry, which will hold our Docker images. These Docker images are built and tagged in our CI script and pushed to this registry. In order to push to this registry we need to setup the registry itself, an IAM user with sufficient permissions and the CI script.
Once you have your AWS CLI and Terraform CLI setup, you will create a new directory for the terraform files.
$ mkdir myapp-terraform
$ cd myapp-terraform
wonderful, now we can start creating and defining our terraform templates.
terraform.tf
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
}
Here we are saying that we want to use AWS as our provider and the version of the AWS provider, pretty simple.
main.tf
provider "aws" {
region = "eu-north-1" # Later please use the region variable
}
In main.tf we actually define the provider which is required by the definition in terraform.tf. Geographically my closest region is eu-north-1 so I'm using it. I advice to choose region based on your resource needs and geographical location. Note that all AWS services/resources my not be available in your region (e.g. CDN).If that's the case, you can define multiple providers with different regions.
variables.tf
variable "name" {
type = string
description = "Name of the application"
default = "myapp"
}
variable "environment_name" {
type = string
description = "Current environment"
default = "development"
}
Here we can define variables for our Terraform templates. These are very handy when you want to have different configurations in dev, staging and production environments. As seen, we define the name of the variable, what type the variable is, description for it and a default value. These variables can be populated or the default values can be replaced with *.tfvars files (e.g. dev.tfvars, stag.tfvars and so on). We will dive in to these later on and how to use environment specific variables.
deploy_user.tf
resource "aws_iam_user" "ci_user" {
name = "ci-deploy-user"
}
resource "aws_iam_user_policy" "ci_ecr_access" {
user = aws_iam_user.ci_user.name
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ecr:GetAuthorizationToken"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Action": [
"ecr:*"
],
"Effect": "Allow",
"Resource": "${aws_ecr_repository.myapp_repo.arn}"
}
]
}
EOF
}
In order to push to the upcoming ECR, we need user with sufficient permissions to do that. Here we are defining a IAM user resource named ci_user and below that we are assigning policies to that user. The policy is in JSON-format and holds the versin and actual statements in an array. On the first statement we are giving specific permissions for our ci_user to get the authorization. On the second statement we are giving all the access rights to our ECR resource (write, read etc.). Read more on here.
ecr.tf
resource "aws_ecr_repository" "myapp_repo" {
name = "${var.environment_name}-${var.name}"
}
This is our final resource till now. This may seem relatively simple, and it is, we are defining an ECR repository for our Docker images. The big part here is how we define the name for the repository. The repository name consists our environment and the application name, so the end result now would be development-myapp since those are our default values in variables.tf. Let's say that we want repositories for staging and production also. We can create corresponding .tfvars-files for them and override the default value which was set in variables.tf. Later on, we will be creating environment specific variable files, but for now I'm leaving them out for simplicity.
Now we have all resources defined and ready to be applied to AWS, but first we will take a look to Terraform workspaces.
Terraform workspaces
Workspaces in Terraform are kind of different environments where you want to deploy your infrastructure. They help you to keep track of the state of different workspaces/environments.
As default we only have one workspace named default. Lets create a now one!
$ terraform workspace new dev
Created and switched to workspace "dev"!
You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.
Wonderful, now we have a new workspace named dev. Now we can try to plan our infrastructure. Note: For the next steps your credentials in the ~/.aws/credentials must be in place. Terraform will respect the AWS_PROFILE environment variable if set, otherwise it will use the default credentials from the earlier mentioned file.
$ terraform plan
What this does, it checks if already existing state file is present, compares your terraform template changes against this state file and prints nicely all the resources which are going to be changed, destroyed or created. When you run above command, you should see that we are creating 3 different resources, ECR repository, IAM user and IAM user policy.
That was the initial planning and checking that our properties for the resources are correct, and we didn't do any mistakes. Now it's time to actually apply this to our AWS environment.
$ terraform apply
Terraform will ask for permissions to perform the listed actions in workspace dev. As Terraform instructs, write yes and your resources will be created in a few seconds! During this, Terraform will create a new folder called terraform.tfstate.d. This folder holds subfolders of your applied workspaces. If you go to terraform.tfstate.d/dev/ you will find a .tfstate-file from there. This file holds the current state of the infrastructure for that particular workspace. This is kinda against the good practises since usually state files are stored somewhere else than locally or in git repository. Read more
In case you ran to an error, check your credentials and syntax of the templates.
Nice, now we have our ECR and IAM user for GitLab CI ready. We can start writing the CI script and start pushing some container images to our newly created repo!
GitLab CI
The GitLab CI script will be just basic yaml file with definitions for jobs, which image to use as a base image since it is Docker/container based CI runtime and many other things!
Things to do first. We need to think what to include to our CI pipeline. Below is a list of stages which we are including to our CI script. You can run multiple jobs parallel in each stage, e.g. we can run security scan and tests at the same time.
- Test
- Build
- Release
- deploy
At this point of the article, I'm going to cover the first three stages and we will revisit the deploy stage later when we have infrastructure ready for it (ECS, RDS etc). Note: .gitlab-ci.yml goes to the same repository where your Elixir + Phoenix application is, and also the CI environment variables go there.
Configuring CI environment
In order to push images to our ECR repository we need to have access to that repository.
Go to your AWS console, navigate to IAM dashboard, go to users and you should see our previously created CI user there. Click the CI user, select security credentials and create access key. This will create access key ID and secret access key for our CI user. Save these two values from the modal view. Other value you need is the ECR repository URL. You can find it by searching for Elastic Container Registry --> repositories and you should see earlier created repository and its URI, copy it.
Go to your GitLab repository and navigate settings --> CI/CD --> expand variables. We are going to add following variables:
AWS_REGIONAWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYDEV_AWS_ECR_URI
Note: Nowadays you can mask CI environment variables, I suggest you to do so for the access key ID, secret access key and ECR URI.
.gitlab-ci.yml
Once we have all the variables in place, we can start writing our script.
stages:
- test
- build
- release
variables:
AWS_DEFAULT_REGION: $AWS_REGION
CONTAINER_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_SHA
AWS_CONTAINER_IMAGE_DEV: $DEV_AWS_ECR_URI:latest
First we define our stages and environment variables on top of the file. The variables are defined on top-level, so they are present in all jobs.
test:
image: elixir:1.10.4-alpine
services:
- postgres:11-alpine
variables:
DATABASE_URL: postgres://postgres:postgres@postgres/myapp_?
APP_BASE_URL: http://localhost:4000
POSTGRES_HOST_AUTH_METHOD: trust
MIX_ENV: test
before_script:
- mix local.rebar --force
- mix local.hex --force
- mix deps.get --only test
- mix ecto.setup
script:
- mix test
Here is our test job which is responsible for running our unit tests. We are using Elixir Alpine based image as a runtime image for this job and defined Postgres as our service, since our tests require database connection. Variables are pretty self explanatory, DB accepts all the connections which are coming to it, MIX_ENV is set to test and all the environment variables for the application itself are present. Before running the actual tests, we install rebar and hex, fetch the dependencies and setup the database. After that we run tests!
build:
stage: build
image: docker:19.03.13
services:
- docker:19.03.13-dind
before_script:
- echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
script:
- docker build -f Dockerfile.production -t $CONTAINER_IMAGE .
- docker push $CONTAINER_IMAGE
only:
- master
On this build job we actually build the Docker image which contains our application. We use official Docker image as our job runtime image and defined docker dind image as our service, since we are going to need some tools from it and we can run "docker in docker". The before_script one-liner will log in to GitLab container registry. On the script we are building our Docker image and pushing it to GitLab container registry (username/project-name). We are accessing our variables/environment variables with $-prefix notation. These can be variables we defined at the top of the CI script or environment variables which is placed in GitLab project settings. Find more on here As sidenote: This job is only run on master-branch.
release_aws_dev:
stage: release
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
dependencies:
- build
image: docker:19.03.13
services:
- docker:19.03.13-dind
before_script:
- apk add --no-cache curl jq python3 py3-pip
- pip install awscli
- echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
- $(aws ecr get-login --no-include-email)
script:
- docker pull $CONTAINER_IMAGE
- docker tag $CONTAINER_IMAGE $AWS_CONTAINER_IMAGE_DEV
- docker push $AWS_CONTAINER_IMAGE_DEV
only:
- master
In release job we are releasing/pushing the recently built image to our ECR repository. For that, we need to define AWS credentials for the job as environment variables for the job runtime. We are using same runtime image and service for this job as in build job. In before_script we are installing needed tools to run AWSCLI, logging in to the GitLab container registry and AWS ECR repository. Once we have logged in, in script we pull the image which we built in the build job, tag it with AWS ECR repository URL which contains the repository name and :latest-tag. After that we push the image to the ECR.
.gitlab-ci.yml
stages:
- test
- build
- release
variables:
AWS_DEFAULT_REGION: $AWS_REGION
CONTAINER_IMAGE: $CI_REGISTRY/$CI_PROJECT_PATH:$CI_COMMIT_SHA
AWS_CONTAINER_IMAGE_DEV: $DEV_AWS_ECR_URI:latest
test:
stage: test
image: elixir:1.10.4-alpine
services:
- postgres:11-alpine
variables:
DATABASE_URL: postgres://postgres:postgres@postgres/yourapp_?
APP_BASE_URL: http://localhost:4000
POSTGRES_HOST_AUTH_METHOD: trust
MIX_ENV: test
before_script:
- mix local.rebar --force
- mix local.hex --force
- mix deps.get --only test
- mix ecto.setup
script:
- mix test
build:
stage: build
image: docker:19.03.13
services:
- docker:19.03.13-dind
before_script:
- echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
script:
- docker build -f Dockerfile.production -t $CONTAINER_IMAGE .
- docker push $CONTAINER_IMAGE
only:
- master
release_aws_dev:
stage: release
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
dependencies:
- build
image: docker:19.03.13
services:
- docker:19.03.13-dind
before_script:
- apk add --no-cache curl jq python3 py3-pip
- pip install awscli
- echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
- $(aws ecr get-login --no-include-email)
script:
- docker pull $CONTAINER_IMAGE
- docker tag $CONTAINER_IMAGE $AWS_CONTAINER_IMAGE_DEV
- docker push $AWS_CONTAINER_IMAGE_DEV
only:
- master
If you did everything correctly, your CI pipeline should pass and your image should show up in the ECR repository! Next up we will actually deploy the application to the cloud!
AWS ECS, RDS, LB and all other folks
Now when we have our images being built and pushed to ECR, it's time to look for the actual deployment. We will defining Terraform resources for database (RDS), container service which is responsible for running the application image (ECS Fargate), load-balancing ((A)LB), networking (subnets, route tables etc.), security groups, some .tfvars-files and output some values once we have applied templates to AWS.
Lets get started!
main.tf
...
resource "aws_vpc" "default" {
cidr_block = "10.0.0.0/16"
tags = {
Environment = var.environment_name
}
}
resource "aws_route_table" "public" {
vpc_id = aws_vpc.default.id
}
resource "aws_route_table" "private" {
vpc_id = aws_vpc.default.id
}
resource "aws_route_table_association" "public_subnet" {
subnet_id = aws_subnet.public.id
route_table_id = aws_route_table.public.id
}
resource "aws_route_table_association" "private_subnet" {
subnet_id = aws_subnet.private.id
route_table_id = aws_route_table.private.id
}
resource "aws_eip" "nat_ip" {
vpc = true
}
resource "aws_internet_gateway" "igw" {
vpc_id = aws_vpc.default.id
}
resource "aws_nat_gateway" "ngw" {
subnet_id = aws_subnet.public.id
allocation_id = aws_eip.nat_ip.id
}
resource "aws_route" "public_igw" {
route_table_id = aws_route_table.public.id
destination_cidr_block = "0.0.0.0/0"
gateway_id = aws_internet_gateway.igw.id
}
resource "aws_route" "private_ngw" {
route_table_id = aws_route_table.private.id
destination_cidr_block = "0.0.0.0/0"
nat_gateway_id = aws_nat_gateway.ngw.id
}
Add VPC (Virtual Private Cloud) resource below the provider. This will be our cloud environment where all other resources will be in. Below it we are configuring a lot of network resources. We are defining route tables for two different subnets (which we will define next), one is exposed to public internet with internet gateway and the other one is private subnet behind NAT gateway. This way our upcoming ECS service can talk to internet (for pulling images from ECR), but no one can get in to it.
subnet.tf
resource "aws_subnet" "public" {
vpc_id = aws_vpc.default.id
cidr_block = "10.0.1.0/24"
tags = {
Environment = var.environment_name
}
}
resource "aws_subnet" "private" {
vpc_id = aws_vpc.default.id
cidr_block = "10.0.2.0/24"
tags = {
Environment = var.environment_name
}
}
resource "aws_subnet" "db_a" {
vpc_id = aws_vpc.default.id
cidr_block = "10.0.3.0/24"
availability_zone = "${var.aws_region}a"
tags = {
Environment = var.environment_name
}
}
resource "aws_subnet" "db_b" {
vpc_id = aws_vpc.default.id
cidr_block = "10.0.4.0/24"
availability_zone = "${var.aws_region}b"
tags = {
Environment = var.environment_name
}
}
resource "aws_db_subnet_group" "default" {
name = "${var.environment_name}-${var.name}-db"
description = "Subnet group for DB"
subnet_ids = [aws_subnet.db_a.id, aws_subnet.db_b.id]
tags = {
Environment = var.environment_name
}
}
Here we have those two subnets, public and private. Also we have defined two subnets for our RDS database instance, which are in different availability zones (AZ). The current upcoming RDS setup requires us to have two availability zones in subnets where it is placed in to. Finally we can create subnet group for the RDS and configure these two DB subnets to it.
security_groups.tf
resource "aws_security_group" "http" {
name = "http"
description = "HTTP traffic"
vpc_id = aws_vpc.default.id
ingress {
from_port = 80
to_port = 80
protocol = "TCP"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group" "https" {
name = "https"
description = "HTTPS traffic"
vpc_id = aws_vpc.default.id
ingress {
from_port = 443
to_port = 443
protocol = "TCP"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group" "egress-all" {
name = "egress_all"
description = "Allow all outbound traffic"
vpc_id = aws_vpc.default.id
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group" "myapp-service" {
name = "${var.environment_name}-${var.name}-service"
vpc_id = aws_vpc.default.id
ingress {
from_port = 4000
to_port = 4000
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
}
resource "aws_security_group" "db" {
name = "${var.environment_name}-${var.name}-db"
description = "Security group for database"
vpc_id = aws_vpc.default.id
ingress {
from_port = 5432
to_port = 5432
protocol = "tcp"
security_groups = [aws_security_group.myapp-service.id]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
lifecycle {
create_before_destroy = true
}
}
Security groups help you to restrict traffic from your resources. E.g. we are only allowing incoming traffic (ingress) to specific ports. Also we can restric the outgoing traffic (egress) to specific IPs etc. These security groups are applied to other resources in our infrastructure (RDS, ECS, Load balancer etc.).
rds.tf
resource "aws_db_instance" "default" {
allocated_storage = var.db_storage
engine = var.db_engine
engine_version = var.db_engine_version
instance_class = var.db_instance_type
name = var.db_name
username = var.db_username
password = var.db_password
availability_zone = var.aws_default_zone
publicly_accessible = false
vpc_security_group_ids = [aws_security_group.db.id]
db_subnet_group_name = aws_db_subnet_group.default.name
tags = {
App = var.name
Environment = var.environment_name
}
}
All the properties are configurable through variables for RDS resource. We are also assigning security group for this resource, and also the subnet group which we defined earlier.
lb.tf
resource "aws_lb" "myapp" {
name = "${var.environment_name}-${var.name}"
subnets = [
aws_subnet.public.id,
aws_subnet.private.id
]
security_groups = [
aws_security_group.http.id,
aws_security_group.https.id,
aws_security_group.egress-all.id
]
tags = {
Environment = var.environment_name
}
}
resource "aws_lb_target_group" "myapp" {
port = "4000"
protocol = "HTTP"
vpc_id = aws_vpc.default.id
target_type = "ip"
health_check {
enabled = true
path = "/health"
matcher = "200"
interval = 30
unhealthy_threshold = 10
timeout = 25
}
tags = {
Environment = var.environment_name
}
depends_on = [aws_lb.myapp]
}
resource "aws_lb_listener" "myapp-http" {
load_balancer_arn = aws_lb.myapp.arn
port = "80"
protocol = "HTTP"
default_action {
target_group_arn = aws_lb_target_group.myapp.arn
type = "forward"
}
}
This is setup for our application load balancer. It will accept basic HTTP requests to port 80 and forward them to our container. If you want to have HTTPS, you must assign certificate to the aws_lb_target_group. You can get one from AWS ACM, but I'm not covering it in this article. For real life production grade systems you want to have SSL/HTTPS always enabled.
ecs.tf
# Role for ECS task
# This is because our Fargate ECS must be able to pull images from ECS
# and put logs from application container to log driver
data "aws_iam_policy_document" "ecs_task_exec_role" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ecs-tasks.amazonaws.com"]
}
}
}
resource "aws_iam_role" "ecsTaskExecutionRole" {
name = "${var.environment_name}-${var.name}-taskrole-ecs"
assume_role_policy = data.aws_iam_policy_document.ecs_task_exec_role.json
}
resource "aws_iam_role_policy_attachment" "ecs_task_exec_role" {
role = aws_iam_role.ecsTaskExecutionRole.name
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}
# Cloudwatch logs
resource "aws_cloudwatch_log_group" "myapp" {
name = "/fargate/${var.environment_name}-${var.name}"
}
# Cluster
resource "aws_ecs_cluster" "default" {
depends_on = [aws_cloudwatch_log_group.myapp]
name = "${var.environment_name}-${var.name}"
}
# Task definition for the application
resource "aws_ecs_task_definition" "myapp" {
family = "${var.environment_name}-${var.name}-td"
requires_compatibilities = ["FARGATE"]
cpu = var.ecs_fargate_application_cpu
memory = var.ecs_fargate_application_mem
network_mode = "awsvpc"
execution_role_arn = aws_iam_role.ecsTaskExecutionRole.arn
container_definitions = <<DEFINITION
[
{
"environment": [
{"name": "SECRET_KEY_BASE", "value": "generate one with mix phx.gen.secret"}
],
"image": "${aws_ecr_repository.myapp_repo.repository_url}:latest",
"name": "${var.environment_name}-${var.name}",
"portMappings": [
{
"containerPort": 4000
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "${aws_cloudwatch_log_group.myapp.name}",
"awslogs-region": "${var.aws_region}",
"awslogs-stream-prefix": "ecs-fargate"
}
}
}
]
DEFINITION
}
resource "aws_ecs_service" "myapp" {
name = "${var.environment_name}-${var.name}-service"
cluster = aws_ecs_cluster.default.id
launch_type = "FARGATE"
task_definition = aws_ecs_task_definition.myapp.arn
desired_count = var.ecs_application_count
load_balancer {
target_group_arn = aws_lb_target_group.myapp.arn
container_name = "${var.environment_name}-${var.name}"
container_port = 4000
}
network_configuration {
assign_public_ip = false
security_groups = [
aws_security_group.egress-all.id,
aws_security_group.myapp-service.id
]
subnets = [aws_subnet.private.id]
}
depends_on = [
aws_lb_listener.myapp-http,
aws_ecs_task_definition.myapp
]
}
There is a lot happening here, but don't get overwhelmed. We are first creating execution role for the ECS task definition (see the comment in the template). After that we define the lob group and the actual ECS cluster. The aws_ecs_task_definition is where all the important configuration happens to you container and environment in and around it. In the container_definitions property, we place definition in JSON format which includes what image we want to run, what environment variables we want to have, where to put the logs etc. Last but not least, we define service where we are going to run that task definition. This basically glues our task definition, cluster, load balancer etc. together.
variables.tf
variable "name" {
type = string
description = "Name of the application"
default = "myapp"
}
variable "environment_name" {
type = string
description = "Current environment"
default = "development"
}
variable "aws_region" {
type = string
description = "Region of the resources"
}
variable "aws_default_zone" {
type = string
description = "The AWS region where the resources will be created"
}
variable "db_storage" {
type = string
description = "Storage size for DB"
default = "20"
}
variable "db_engine" {
type = string
description = "DB Engine"
default = "postgres"
}
variable "db_engine_version" {
type = string
description = "Version of the database engine"
default = "11"
}
variable "db_instance_type" {
type = string
description = "Type of the DB instance"
default = "db.t3.micro"
}
variable "db_name" {
type = string
description = "Name of the db"
}
variable "db_username" {
type = string
description = "Name of the DB user"
}
variable "db_password" {
type = string
description = "Name of the DB user"
}
variable "ecs_fargate_application_cpu" {
type = string
description = "CPU units"
}
variable "ecs_fargate_application_mem" {
type = string
description = "Memory value"
}
variable "ecs_application_count" {
type = number
description = "Container count of the application"
default = 1
}
Here are the variables which we are uring across the Terraform templates.
environment/dev.tfvars
environment_name = ""
aws_region = ""
db_name = ""
db_username = ""
db_password = ""
ecs_fargate_application_cpu = "256"
ecs_fargate_application_mem = "512"
ecs_application_count = 1
Here you can fill the values for the variables. Note that you can create multiple workspaces e.g. dev, stag and prod, and also create .tfvars-files accordingly dev/stag/prod.tfvars. I'll show you soon how to use them.
outputs.tf
output "load_balancer_dns" {
value = aws_lb.myapp.dns_name
}
Here we are just printing the URI out of our load balancer. We can use this to access our application from browser (or if you application is an API, then in Postman or other equivalent software).
deploy_user.tf
resource "aws_iam_user_policy" "ecs-fargate-deploy" {
user = aws_iam_user.ci_user.name
policy = <<POLICY
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"ecs:UpdateService",
"ecs:UpdateTaskDefinition",
"ecs:DescribeServices",
"ecs:DescribeTaskDefinition",
"ecs:DescribeTasks",
"ecs:RegisterTaskDefinition",
"ecs:ListTasks"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
POLICY
}
Last but not least, we assign user policy to our IAM CI user. This way we can deploy a new version of the application to ECS.
Now we can start applying this to AWS.
$ terraform plan --var-file=environment/dev.tfvars
First you want to plan the current changes your infrastructure. This will catch any typos and configuration mistakes in your templates. If it goes through without any problems, you can start applying.
$ terraform apply --var-file=environment/dev.tfvars
Terraform will ask you do you want to apply these changes, go through the changes and if they look good, hit Terraform with a yes answer. Now it will take some time to create all the resources to AWS. Grab a cup of coffee/tea/whatever and watch.
Now we are pretty much done for the infrstructure part. Next we will update our CI script and add the deploy job to it which will update the ECS service with newly built application Docker image from ECR.
Update .gitlab-ci.yml
Below you can see the changes we must do in order to add deploy job to our CI pipeline.
.gitlab-ci.yml
stages:
- test
- build
- release
- deploy
...
...
...
deploy_aws_dev:
stage: deploy
dependencies:
- release_aws_dev
image: alpine:latest
variables:
AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
before_script:
- apk add --no-cache curl jq python3 py3-pip
- pip install awscli
script:
- aws ecs update-service --force-new-deployment --cluster development-myapp --service development-myapp-service
only:
- master
First add deployment stage on top of the file and then we add the job on the bottom of the file. In before_script we install needed tools and AWSCLI. On script we make the actual deployment. This tells to update the service from the defined cluster.
Conclusions
If you are developing a small scale application, this kind of infrstructure may be an "overkill" for it, but for medium to large sized applications this would be a good starting point. Elixir + Phoenix itself can handle concurrently large amount of requests, but ability to be able to scale underlaying infrastructure with Terraform is a feature you want to have when your application grows.
Things to do better:
- HTTPS/SSL certificate and domain for the application - This is pretty straight forward, but I decided to leave it out for now. It would require ACM resource with output of the domain validation options for setting CNAME record to you domain DNS.
- Better Terraform variable usage - We could map multiple subnet AZ to single variable and use Terraform's functions to map those values.
- VPC endpoints - Instead of accessing ECR images through NAT from ECS, we could define VPC Endpoints for ECR, S3 and CloudWatch. This way we could keep all the traffic on the private network.
This was a fun little project and gives you a nice starting point for you infrastructure and CI pipelines in GitLab for Elixir + Phoenix. This article came out less "elixiry" than I wanted, this same template can be applied for other languages as well.
Any feedback is appreciated!
Top comments (13)
Hi mate,
thanks for the great and educational article :)
Could you pls help with this issue by any chance? :)
terraform plan
Error: Reference to undeclared resource
on deploy_user.tf line 24, in resource "aws_iam_user_policy" "ci_ecr_access":
24: "Resource": "${aws_ecr_repository.myapp_repo.arn}"
A managed resource "aws_ecr_repository" "myapp_repo" has not been declared in
the root module.
This basically means that the ECR repository you are referring to in the user policy does not exist. Check the naming of the ECR repository resource.
I fixed one typo from the article, there was unnecessary "_url" in the ECR naming, sorry about that!
Thanks for getting back to me, mate :)
Sorry that asking such silly question. Could I fix it easily by adding that ECR repo onto AWS before runnig Terrafrom commands? Unfortunatelly, I haven't touched this ECR service, yet :)
Oh, that's tottaly fine, my typos are beond my understanding lol
Well, you can add it manually through AWS console if you want to. Also, you can just rename the resource and Terraform will destroy the old one and create a new one with the new name. :)
Thanks bro, sorry, I didn't correct that typo earlier, I thought it's gonna be a next iteration of the following article heh :)
Couldn't trace out this error yet when pusshing code to my GitLab repo. Have you seen it before?
$ which elixir
/usr/local/bin/elixir
$ mix --version
Erlang/OTP 22 [erts-10.7.2.4] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1]
Mix 1.10.4 (compiled with Erlang/OTP 22)
$ ln -s /usr/local/bin/mix /usr/bin/mix
$ mix local.rebar --force
Is this output on the CI or on your local machine? If local, what OS are you using?
Yeah, on CI. I've already pushed that code to my repo :)
pastebin.com/vdbDW8Um
gitlab.com/organicnz/myapp-terraform
I see your problem! Please place the
.gitlab-ci.ymlto repository which contains your Elixir + Phoenix application. It should be placed to the root of the repository.Second thing, never push the state files publicly to any platform. They should be ignored with
.gitignore, crypted with git-crypt (or other encrypting tool) or the best solution would be to use remote state.Would it be appropriate if I use your Elixir application for the CI github.com/hlappa/microservice_exe... or anyone? :)
Jeez, that's right I completely forgot to sanitise that area, already added .gitignore file then will try to figure out how to push it to the remote state on AWS :)
Thanks for the fantastic write up!
1) FYI as of 1 October 2020 the GitLab free tier was reduced to 400 minutes of CI/CD.
2) The initial CI/CD run failed for me with below message in the test stage. I haven't figured out the fix for this yet.
3) Further down in the tutorial (code block for rds.tf) you've got:
but you haven't defined it in your variables.tf
Sorry for a late reply to you comment, good points!
1) Article is now updated. I didn't know GitLab reduced the minutes from 2000 to 400, which is kinda sad :(
2) Make sure you have Postgres as a service for the test job, the DB url is correctly setup and check your
config/test.exthat is has proper configuration. Seems like when establishing the connection to DB fails.3) Added to
variables.tf! :)Hi @Aleksi! Thanks for the article.
Please update the ecs task with the DATAABASE_URL envnvar.
This works.
{"name": "DATABASE_URL", "value": "ecto://${var.db_username}:${var.db_password}@${aws_db_instance.default.address}:${aws_db_instance.default.port}/${var.db_name}"}