TLDR
Having a continuous integration / continuous delivery (CI/CD) pipeline set up for each of our git repositories is an essential part of Mage’s software development life cycle. It allows us to greatly speed up the process of testing, building, and deploying our application without sacrificing quality assurance.
Outline
- Introduction to CI/CD
- Importance of CI/CD in software development
- CircleCI and its alternatives
- How Mage uses CircleCI for our CI/CD pipeline
- CircleCI configuration
Introduction to CI/CD
The “CI” stands for “continuous integration”, and it’s the practice of regularly building and testing code after each code change is merged to the main branch. Different tests can be run depending if code is committed to a secondary branch or the main branch.
The “CD” stands for either “continuous delivery” or “continuous deployment”. Continuous delivery and continuous deployment both build on top of continuous integration and enable automatic deployments of code changes to testing/staging and production environments.
Delivering your app to the interwebs. (Source: Giphy/TikTok)
However, continuous delivery requires a manual step, such as an approval, before deploying to production, whereas continuous deployment automates the production deployment to completion without an approval step, assuming all tests and builds have successfully completed.
Importance of CI/CD in software development
Using a CI/CD pipeline automatically tests new code commits at frequent intervals so bugs or breaking changes can be caught quickly and rectified before being deployed to production. The release process is much quicker, easier, and safer once the CI/CD pipeline is already set up, so it makes your team’s software development life cycle much more efficient.
CircleCI and its alternatives
CircleCI flow (Source: https://circleci.com/docs/2.0/about-circleci/)
CircleCI is currently one of the most popular CI/CD tools on the market, and it supports several widely used programming languages, including Python, JavaScript, Java, Go, and Ruby on Rails. It also supports integration with both GitHub and Bitbucket.
Though CircleCI is the tool that Mage uses and that we will go into greater detail in the rest of this article, there are several other suitable CI/CD tools available, such as Jenkins, TeamCity, or Travis CI.
As stated in our Startup tech stack article, we previously used Travis CI but migrated our CI/CD pipeline to CircleCI because it allowed better control over configuring our pipeline (e.g. setting up manual approvals for certain jobs before the next job can run), was less restrictive in terms of number of concurrent jobs and users, and was a much better value for us (ended up costing five times less than Travis CI).
How Mage uses CircleCI for our CI/CD pipeline
Our CircleCI backend workflow
We have different CircleCI workflows for each of our services, but we’ll be discussing the different parts of our backend workflow below. A workflow defines a collection of jobs and the order in which they are run, and a job is a collection of steps. All steps in a job are executed in a single Docker container or virtual machine (VM).
Github integration
We host all of our code repositories in GitHub, so we integrate CircleCI with all of our services except the front-end service, which we use Vercel for. We discuss how we use Vercel in the “Infrastructure” section of our Tech stack article.
When working on secondary branches, code commits pushed to the remote branch on GitHub will only build a development environment in a Docker container and run tests, but not make any deployments.
Testing/Debugging
In addition to the automated testing that CircleCI provides on every code commit pushed to the remote branch, CircleCI also allows us to SSH into test instances to check our code or test environment for debugging purposes.
Eliminating bugs in your pipeline. (Source: Giphy)
This SSH feature can be found by clicking on the “Rerun” button in the top right corner of the CircleCI job page and selecting “Rerun Job with SSH”:
Database migrations
On the main branch after all tests have successfully passed, CircleCI runs our database migrations for our MySQL database hosted on Amazon Relational Database Service (RDS). This synchronizes the production database state with any updated models from our Django backend.
Staging environment
After successfully running the database migrations, CircleCI deploys our backend application to a staging environment in Amazon Elastic Beanstalk and also runs some smoke tests automatically. With the staging environment deployed, we can run additional manual tests and confirm that any new or updated APIs are working properly.
Notifications
CircleCI can send emails to members of a CircleCI project’s team whenever a workflow succeeds or fails to complete. There is also a Slack integration that can send Slack messages at any step during the workflow. We have a message sent to a specific Mage Slack channel whenever any manual approvals are required to remind us to go back into the CircleCI UI.
Manual approvals
We require a manual approval in CircleCI for initiating any deployments to production. This allows us to do any additional custom testing in the staging environment first, and it also gives us control over the timing of our deployments.
Manual approvals actually require a team member to go into the CircleCI UI and click an “Approve” button, like in the screenshot below:
Building Docker images
As part of our CircleCI workflow, we can also create custom Docker images using our own Dockerfiles.
We use AWS Lambda to execute and quickly complete complex data processing tasks asynchronously. In order to configure our AWS Lambda functions, we need to build custom Docker images and push them to Amazon Elastic Container Registry (ECR). This is done automatically in CircleCI after approving the manual approval hold.
Production deployments
In addition to updating the AWS Lambda functions, simultaneous CircleCI jobs will be run for deploying our application and backend workers to production in Amazon Elastic Beanstalk.
Deployment rollbacks
There may be situations where we want to rollback a deployment. For instance, if we already deployed our application to production but want to revert back to a previous version because we discovered an uncaught bug, we may be able to do that using CircleCI.
In order to rerun an individual job in a workflow after it has already successfully completed, CircleCI doesn’t have an explicit feature that allows us to do this from the UI. However, as a workaround, we can use the “rerun the job with SSH” feature like in the “Testing/Debugging” section above. We simply go back to the workflow for that particular code commit, and rerun the individual deployment job.
Note that the rerun deployment job should be able to reference the relevant Docker image with a unique version tag (e.g. a commit SHA hash value) if the job is deploying an application using Docker containers, which is the case for our API service.
CircleCI configuration
A config.yml file using YAML syntax is required for defining the steps, jobs, and workflows in CircleCI. We’ll go over some of the different parts of a config.yml file, but the best place for details on the different configuration keys would be the CircleCI documentation.
Version
A minimum CircleCI version of 2.1 is required for several different features, so it’s not recommended to use a version lower than that.
version: 2.1
Orbs
Orbs are convenient packages of configuration elements (i.e. jobs, commands, executors) provided by CircleCI. There are many orbs available in CircleCI’s orbs registry.
Some of the orbs used by Mage include the Slack, AWS Elastic Beanstalk, AWS CLI, and AWS ECR orbs. The Slack orb allows CircleCI to send messages to one of our Slack channels, while the AWS orbs allows us to easily run CLI or other commands specific to those tools.
orbs:
slack: circleci/slack@4.3.1
aws-elastic-beanstalk: circleci/aws-elastic-beanstalk@1.0.2
aws-cli: circleci/aws-cli@1.4.0
aws-ecr: circleci/aws-ecr@6.15.3
Jobs
Most of the sections under “How Mage uses CircleCI for our CI/CD pipeline” above represent a single job in Mage’s backend workflow. For example, there is a job for running tests, a job for deploying to a staging environment, a job for building the AWS Lambda Docker image and deploying it to AWS ECR, etc.
Each job has a number of steps that get executed in a single Docker container or VM. In our CircleCI jobs, we generally use Docker containers whenever possible because they can spin up quicker than booting up a VM, and our jobs aren’t so resource-intensive that they require being run in a VM.
If your jobs only require a minimal amount of CPU and memory, a small Docker container uses less resources than a medium VM (the smallest size available in CircleCI), resulting in less CircleCI credits being used per minute.
The slack/on-hold job provided by a CircleCI orb has the following steps:
The CircleCI docs have several examples of how the configuration should look like inside the config.yml file for jobs.
Workflows
There is a lot of customization regarding how a CircleCI workflow can run. Besides defining the order that jobs should run in, we can specify which branches we want certain jobs to run on, fan-out to run multiple jobs concurrently to save time, or require several jobs to successfully complete before a specific job can proceed.
We showed a diagram of Mage’s backend workflow earlier in the article at the very beginning of the “How Mage uses CircleCI for our CI/CD pipeline” section. This is just one of many ways to utilize the power of CircleCI workflows.
Closing thoughts
No matter which CI/CD tool you decide to use, the important thing is just having a CI/CD pipeline in place to begin with and then building off of it. Migrating to another platform can always happen later if absolutely necessary.
Relax while your pipeline takes care of the work for you. (Source: Giphy)
Top comments (8)
Great read on CI/CD!
Thx for this! This is really what I wanted. Helped A LOT.
Can I translate in Korean this article? If you don't mind, I wanna share this awesome information in Korean. Surely, There will be a link directing to this original one.
Glad to hear that our article was helpful to you! We hope to have our content translated in the future but do not mind if you want to translate for now with a link.
Using good CI/CD tool makes shipping code faster and safer!
I’ve always wondered how startups configure their CI/CD pipeline. A good setup saves a ton of dev time.
CI/CD has always been so useful at eliminating bugs! Great read for anyone curious about how it works.
Super cool. Happy to have your feedback on Qovery - the easiest way to deploy your app from Github and Gitlab
Thanks for reading the article, @rophilogene ! Qovery looks like a cool app. Lots of useful guides and docs on the site as well. Just wondering, if a user wants to deploy their app onto AWS, is Qovery intended to be much simpler than using something like Elastic Beanstalk?