CircleCI + AWS How to create CI/CD pipeline from scratch Part 2 - Setup and update servers

#aws #devops

After all previous steps, you've got the images that are stored inside the ECR and the script that automatically build
them. In this part of the tutorial we will:

setup ECS clusters for development and production environments;
add deployment commands to the CircleCI script.

Let's do it!

Initialize ECS cluster

Creating of cluster consists of three steps:

Create an empty cluster with a VPC
Define the task that will launch the selected container
Add service that will launch and maintain the desired count of ec2 instances with the previously defined task

Create cluster

Let's start with the definition of cluster:

An Amazon ECS cluster is a logical grouping of tasks or services.

Roughly saying, clusters define the scope and set of rules for the launched tasks. To create one, follow the steps below:

Select EC2 Linux + Networking you need two clusters one for development and one for master branches
Select 1 On Demand t2.micro instance(or other types of ec2 instances), other configurations by default

Create cluster
For the networking section, I recommend to use the default parameters too. It will create a new VPC with
security group allowing income traffic
to the 80 PORT.

Yes, that's all, you should create one for the development and one for the production.

Define task

Tasks are used to run Docker containers in Amazon ECS.

Select EC2 launch type compatibility
Choose the name for the task definition(or family, like sometimes it's called)
For the as the task role choose the role with the AmazonECSTaskExecutionRolePolicy policy that we previously created
Select the memory limit, for example, 128 MB if your server is not going to handle a lot of complex requests
And finally, add a container. On this tab we are interested in the following fields:
- Standard -> Image - initial image for the task should be copied from the ECR looks like 845992245040.dkr.ecr.us-west-2.amazonaws.com/simple_plug_server:master_latest
- Standard -> Port mappings - associate host 80 with the port which is using our application and will be defined next
  Add container to a task and configure the port mapping

Advanced container configuration -> ENVIRONMENT -> Environment variables - define the variable with the name PORT and desired value for example - 4100. This value must be used in the port mapping as the container port
Set up environment variables

Great! You've created the first revision of the task definition. Of course, these tasks can be launched right inside the cluster, but we will use services to simplify updating tasks revisions. Let's add them.

Add service

An Amazon ECS service enables you to run and maintain a specified number of instances of a task definition simultaneously in an Amazon ECS cluster.

To create a service just click on the Create button on the Services and fill the form:

Select EC2 in the Launch type selector, because we are deploying our tasks on EC2 instances
In the Task Definition select the task and revision that you are created earlier
In the Cluster select the cluster where you want to define a service. When you are creating service from a cluster this field will be already selected
Service type - REPLICA
Number of tasks - 1 because we do not care about scaling for now.
Other setting set by default

After repeating all steps for the development and production cluster, you've got two EC2 instances with running applications. They are available by Public DNS or IP.

Now let's go to the final part of this tutorial. 🚀

Add deployment scripts

With an official orb aws-ecs you can make this very easy. We already added all necessary environment variables in the second part of this tutorial so you should only modify the circleci config.

Result version of the .circleci/config.yml

version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@6.9.1
  aws-ecs: circleci/aws-ecs@1.2.0
jobs:
  test:
    docker:
      - image: elixir:1.10
        environment:
          MIX_ENV: test
    working_directory: ~/repo
    steps:
      - checkout
      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: mix deps.get
      - run: mix deps.compile
      - run: mix test
      - store_test_results:
          path: _build/test/lib/simple_plug_server
workflows:
  version: 2
  test-build-deploy:
    jobs:
      - test
      - aws-ecr/build-and-push-image:
          repo: "simple_plug_server"
          tag: "${CIRCLE_BRANCH}_${CIRCLE_SHA1},${CIRCLE_BRANCH}_latest"
          requires:
            - test
          filters:
            branches:
              only:
                - master
                - development
      - aws-ecs/deploy-service-update:
          name: deploy-development
          requires:
            - aws-ecr/build-and-push-image 
          family: "simple-plug-server-development"
          cluster-name: "SimplePlugServer-development"
          service-name: "sps-dev-serv"
          container-image-name-updates: "container=simple-plug-server-development,tag=${CIRCLE_BRANCH}_${CIRCLE_SHA1}"
          filters:
            branches:
              only:
                - development
      - approve-deploy:
          type: approval
          requires:
            - aws-ecr/build-and-push-image
          filters:
            branches:
              only:
                - master
      - aws-ecs/deploy-service-update:
          name: deploy-production
          requires:
            - approve-deploy
          family: "simple-plug-server-production"
          cluster-name: "SimplePlugServer-production"
          service-name: "simple-plug-server-production"
          container-image-name-updates: "container=simple-plug-server-production,tag=${CIRCLE_BRANCH}_${CIRCLE_SHA1}"
          filters:
            branches:
              only:
                - master

In this file was added three new jobs. Two aws-ecs/deploy-service-update respond for the updating respective services in the clusters and approve-deploy that's waiting for confirmation before the last step for the master branch. For different branches, flows will be a little different. It can be achieved by using parameter filters in job definitions, where you can specify for which branches or git tags launch jobs.

aws-ecs/deploy-service-update job configuration

I would like to tell you about the parameters for the job aws-ecs/deploy-service-update:

name - the name is used to make jobs in a workflow more human-readable. I am sure you would agree that's deploy-production looks much more clearer than aws-ecs/deploy-service-update.
requires - used to define the order of jobs execution, namely the previous job that must be finished successfully.
family - there you should write the name of the task definition(Define task) that you used when you created the task
cluster-name - it's pretty obvious - the name of the desired cluster where all magic happens
service-name - the name of the service that's managing tasks inside the previously mentioned cluster
container-image-name-updates - updates the Docker image names and/or tag names of existing containers that had been defined in the previous task definition
- container - the name of the container that you used when you added the container to the task(circled in blue on the screenshot Add container to a task and configure the port mapping)
- tag - one of the tags that you are defined in the aws-ecr/build-and-push-image job, in this example it's a ${CIRCLE_BRANCH}_${CIRCLE_SHA1}

And that's all 🎉. When you push your branch with the new circleci config and start to work you will see something like that.

Service creation

I hope that this tutorial was helpful and was not wasted your time. If you have any question and problems feel free to ask me about it in the comments. 👋