How To Automate Your Deployments To AWS EC2 Using CircleCI And Ansible

Abuchi Kingsley — Sat, 21 Oct 2023 21:41:13 +0000

Introduction

In today's world of fast paced releases and distributed teams working across flexible hours, the importance of automated deployments cannot be overstated. In this tutorial I will guide you through the steps to configure CircleCi and Ansible for automated deployments to AWS EC2.

Prerequisite

To get the most out of this tutorial and follow along it will be nice to have:

Basic programming experience and be comfortable with the command line.
A Github account and access to command line with git installed.
An AWS account with EC2 server set up.
A CircleCI account.
A code editor.

This tutorial will focus on configuration and deployment only, we won't be setting up new servers. I will be using a backend server I wrote a while ago. You can find the starter code here. Fork and clone the repository. Switch to the starter branch to follow along.

CircleCI Set-up

Head over to circleci.com and login. Ensure you are within your personal organization. On the "Projects" page search for aws_ec2_auto_deploy repo and just click "Set Up Project". Select "Use the . circleci/config. yml in my repo" option and set up project. Set up your project environment variables by copying the default values in .env.example file over to Project Settings > Environment Variables on CircleCI project dashboard. Now, let's get down to brass tacks.

Writing `config.yml`

On your editor, open the config.yml file inside .circleci folder.

CircleCI uses YAML files to configure pipelines which is usually made up of the following blocks of YAML codes:

YAML Block	Description
`version`	This specifies the version of CircleCI API you wish to use
`commands`	Contains list of reusable commands
`orbs`	Makes it possible to use certain prewritten functionalities in a job
`jobs`	List of jobs
`workflow`	Defines the order to run jobs

Before deploying your code to the servers, it is good practice to always ensure that all tests are passing. Update your config.yml file with the code below.

version: 2.1

jobs:
  test:
    docker:
      - image: cimg/node:16.17.0
      - image: redis
        name: redis
      - image: mysql:8.0
        environment:
          MYSQL_ROOT_PASSWORD: 12345
          MYSQL_DATABASE: lend
          MYSQL_USER: abuchikings
          MYSQL_PASSWORD: 12345
    steps:
      - checkout
      - restore_cache:
          keys: [packages]
      - run:
          name: Test MYSQL connection
          command: |
            for i in `seq 1 30`;
            do
              nc -z 127.0.0.1 3306 && echo Success && exit 0
              echo -n .
              sleep 1
            done
            echo Failed waiting for MySQL && exit 1

      - run:
          name: Run tests
          command: |

            npm install
            npm run migrate:latest
            npm run test

      - save_cache:
          paths:
            - "./node_modules"
          key: packages

workflows:
  default:
    jobs:
      - test

Here is a break down of what is happening in the code above:

The jobs keyword indicates that the code block is going to contain jobs.
The very first job is named test and the docker keyword tells CircleCI that this job is going to be executed within a docker environment.
cimg/node is CircleCI's official nodejs image on docker hub. Node version 16.17.0 is used for this project. MySQL 8.0 database and redis are required to test the application.
steps: As its name suggests defines the steps required to run a specific job.
checkout: This checks out the source code from your version control repository.
restore_cache: Restores cached dependencies. Caching is a mechanism that allows you to save and reuse dependencies (e.g., npm packages) to speed up the build process. In this case, it restores cached npm packages.
run: This step runs a series of commands. It's responsible for running the actual CI tasks.
name: This provides a descriptive name for this step, indicating that the following commands are related to running tests.
command: Specifies the commands to be executed. The | symbol is used to define a multi-line command block, allowing the execution of multiple commands within this step. The first command utilizes nc (netcat) utility to check if a MySQL database server running on the localhost (127.0.0.1) and listening on port 3306 is available. The second command runs migration and automated tests on the code.
save_cache: This is a built-in CircleCI command used to save cached dependencies or artefacts. It ensures that the specified files or directories are cached for future builds.
workflows: This section defines one or more workflows. A workflow is a series of jobs that need to be executed in a particular order or under specific conditions.

Now you can commit your changes and push them to remote repository. Head over to CircleCI dashboard to see your pipeline run.

Next update your config.yml with the setup_ansible job below. Remember to to also update your workflow as shown in the code snippet below.

  setup_ansible:
    docker:
      - image: amazon/aws-cli
    steps:
      - checkout
      - run:
          name: Install dependencies
          command: |
            yum install -y tar gzip
      - run:
          name: Add Backend IP To Ansible
          command: |
            backend_ip=$(aws ec2 describe-instances \
            --query "Reservations[*].Instances[*].PublicIpAddress" \
            --filters "Name=tag:Name,Values=ec2_auto_deploy_test" \
            --output text)
            echo "$backend_ip" >> ~/project/.circleci/ansible/inventory.txt
            cat ~/project/.circleci/ansible/inventory.txt
      - persist_to_workspace:
          root: ~/
          paths:
            - project/.circleci/ansible/inventory.txt

workflows:
  default:
    jobs:
      - test
      - setup_ansible:
          requires: [test]

Remember to change ec2_auto_deploy_test to the name of your EC2 server on AWS.

The setup_ansible job uses aws-cli to get the IP address of an EC2 server with name tag ec2_auto_deploy_test. The IP address is saved to the inventory.txt file within the Ansible folder and persisted to the workspace for use by the next job. Utilities tar and gzip are required to persist file to workspace.

Before pushing your changes to github, head over to the project page on CircleCI => Project Settings => Environment Variables and add the following environment variables:

AWS_SECRET_ACCESS_KEY
AWS_DEFAULT_REGION
AWS_ACCESS_KEY_ID

which can be obtained from your aws account.

The final job configuration on the config.yml file will be responsible for executing an Ansible playbook that uses ssh to access the server, pulls code from github and builds an updated docker image. See code below.

  deploy_server:
    docker:
      - image: python:3.11.6-alpine3.18
    steps:
      - checkout
      - add_ssh_keys:
          fingerprints: ["9f:66:26:48:fd:251:62:5d:73:c4:d5:90:2b"]
      - attach_workspace:
          at: ~/
      - run:
          name: Install dependencies
          command: |
            apk add --update ansible openssh-client
            eval $(ssh-agent -s)
      - run:
          name: Deploy Server
          command: |
            cd ~/project/.circleci/ansible
            echo "Contents  of the inventory.txt -------"
            cat inventory.txt
            ansible-playbook -i inventory.txt deploy-server.yml

On your CircleCI project settings page, go to SSH Keys, add a new ssh key by copying and pasting the ssh key for your EC2 server. This will generate a key fingerprint. Copy the SSH key fingerprint and replace fingerprints on the deploy_server job.

Also update your workflow with the following code:

workflows:
  default:
    jobs:
      - test
      - setup_ansible:
          requires: [test]
      - deploy_server:
          requires: [setup_ansible]

Ansible Set-up

Ansible is a powerful open-source automation tool and configuration management system that streamlines and simplifies the automation of routine IT tasks and the management of complex systems. It provides a framework for IT professionals to define and execute automation tasks through human-readable playbooks.

With Ansible, you can automate the provisioning, configuration, deployment, and maintenance of servers, applications, and various infrastructure components. You should check out Ansible docs for more on this interesting automation tool.

Writing A Playbook

To automate Ansible, we use simple YAML files called playbooks to define automation tasks. The following shows the tree structure for the Ansible folder which contains the playbook to be executed.

├── ansible
│   ├── ansible.cfg
│   ├── deploy-server.yml
│   ├── inventory.txt
│   └── roles
│       └── deploy
│           └── tasks
│               └── main.yml
└── config.yml

Here is a breakdown of each file and its function:

ansible.cfg: A configuration file used to customize the behaviour of Ansible. This allows you to override built-in configuration settings like SSH connection options, default inventory file, define remote user privileges and so on.
deploy-server.yml: This is an Ansible playbook. It defines a set of tasks or a task to be executed on remote hosts. In this project, I used roles to define playbook tasks.
inventory.txt: This serves as a critical component in Ansible for defining the hosts and groups of hosts that Ansible will manage. It is a configuration file that specifies the remote servers or systems where Ansible tasks and playbooks will be executed. In this file, IP addresses of the servers or machines to be managed by Ansible are listed.
roles: Roles are a way to organize and structure tasks, handlers, variables, and other content for better code reuse and maintainability. This contains the tasks to be executed in playbooks. These tasks are defined within the main.yml file.

Update deploy-server.yml with the following code:

---
- name: "Deploy Play"
  hosts: web
  user: ubuntu
  gather_facts: false
  vars:
    - ansible_python_interpreter: /usr/bin/python3
    - ansible_host_key_checking: false
    - ansible_stdout_callback: yaml
  roles:
    - deploy

name: This is a user-defined name for this playbook that makes it easy for us to understand its function.
hosts: This specifies the target hosts or hosts group on which the tasks in this playbook will be executed. In this case it is set to web. When you open the inventory.txt file, it contains a single line, [web]. This defines a host group called web and all IP Addresses listed under [web] are the hosts or remote machines we wish to manage with Ansible. To add an IP address to the inventory we are using aws-cli in the setup_ansible job to pull the IP Address of our EC2 server and dynamically update the inventory file.
user: This specifies the remote user that Ansible should use when connecting to the target hosts. In this case it is ubuntu. You may need to change that if your ssh user is different. gather_facts: This tells Ansible whether to collect facts about the target hosts, such as hardware details, network configuration, and more. Here it is set to false to improve playbook execution speed.
vars: This defines variables to be used in the playbook. ansible_python_interpreter specifies the python intepreter Ansible should use when running jobs on the host. ansible_host_key_checking is set to false to disable host key checking. Host key checking is a prompt that comes up on your command line when you are executing ssh login to a server with a key for the first time. This is a security measure to ensure the remote host identity.
roles: This specifies which roles should be included in this job. In this case, it's a role named deploy. When the playbook is executed, Ansible will look for a directory named "deploy" within the roles path and execute the tasks and configurations defined in that role. Let's write the tasks for the deploy role.

Copy the code below to main.yml within the deploy/tasks role folder.

---
- name: Perform Git Pull
  become: false
  git:
    repo: git@github.com:AbuchiKings/aws_ec2_auto_deploy.git # Replace with github repo you prefer
    dest: ~/apps/aws_ec2_auto_deploy
    accept_hostkey: yes
    clone: false
    version: main # Replace with the desired branch or tag

- name: Run Docker build
  become: true
  shell: 
    chdir: apps/aws_ec2_auto_deploy
    cmd: make "{{ item }}"
  loop: 
      - down
      - up-d

The first task uses Ansible's git module to perform a git pull from the project's git repository.

become is a boolean field that tells Ansible to perform a task as a root user.
dest is the destination or directory where the fetched items should be added.
To prevent a fresh cloning of the repo, clone is set to false.
version defines the branch to pull from.

In the second task, the make command is used on two different arguments defined inside a Makefile in the project root folder:

down which maps to docker-compose -f docker-compose.yml -f docker-compose.prod.yml down takes down the current running container.
up-d which maps to docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d --build starts up the container with a new build in a detached mode.

Finally, commit your changes and push to remote repository.

Conclusion

This is by no means means an exhaustive tutorial on CircleCi and Ansible. There are a wide range of automation tasks you could perform with Ansible. Nonetheless, I hope you find this helpful. Thank you for your time.

DEV Community: Abuchi Kingsley