loading...
Cover image for CI/CD Pipeline: A Gentle Introduction

CI/CD Pipeline: A Gentle Introduction

markoa profile image Marko Anastasov Updated on ・4 min read

Do you want to ship bug-free code at high velocity? A fast and reliable CI/CD pipeline is crucial for doing that sustainably over time.

What is a CI/CD pipeline?

A CI/CD pipeline helps you automate steps in your software delivery process, such as initiating code builds, running automated tests, and deploying to a staging or production environment. Automated pipelines remove manual errors, provide standardized development feedback loops and enable fast product iterations.

What do CI and CD mean?

CI, short for Continuous Integration, is a software development practice in which all developers merge code changes in a central repository multiple times a day. CD stands for Continuous Delivery, which on top of Continuous Integration adds the practice of automating the entire software release process.

With CI, each change in code triggers an automated build-and-test sequence for the given project, providing feedback to the developer(s) who made the change. The entire CI feedback loop should run in less than 10 minutes.

Continuous Delivery includes infrastructure provisioning and deployment, which may be manual and consist of multiple stages. What's important is that all these processes are fully automated, with each run fully logged and visible to the entire team.

Elements of a CI/CD pipeline

A CI/CD pipeline may sound like overhead but it really isn't. It's essentially a runnable specification of the steps that need to be performed in order to deliver a new version of a software product. In the absence of an automated pipeline, engineers would still need to perform these steps manually, and hence far less productively.

Most software releases go through a couple of typical stages:

CI/CD pipeline stages

Failure in each stage typically triggers a notification—via email, Slack, etc.—to let the responsible developers know about the cause. Otherwise notifications are usually configured to be sent to the whole team after each successful deploy to production.

Source stage

In most cases a pipeline run is triggered by a source code repository. A change in code triggers a notification to the CI/CD tool, which runs the corresponding pipeline. Other common triggers include automatically scheduled or user-initiated workflows, as well as results of other pipelines.

Build stage

We combine the source code and its dependencies to build a runnable instance of our product that we can potentially ship to our end users. Programs written in languages such as Java, C/C++, or Go need to be compiled, whereas Ruby, Python and JavaScript programs work without this step.

Regardless of the language, cloud-native software is typically deployed with Docker, in which case this stage of the CI/CD pipeline builds the Docker containers.

Failure to pass the build stage is an indicator of a fundamental problem in the configuration of our project and it's best to address it immediately.

Test stage

In this phase we run automated tests to validate the correctness of our code and the behavior of our product. The test stage acts as a safety net that prevents easily reproducible bugs from reaching the end users.

The responsibility of writing tests falls on the developers, and is best done while we write new code in the process of test- or behavior-driven development.

Depending on the size and complexity of the project, this phase can last from seconds to hours. Many large-scale projects run tests in multiple stages, starting with smoke tests that perform quick sanity checks to end-to-end integration tests that test the entire system from the user's point of view. A large test suite is typically parallelized to reduce run time.

Failure during the test stage exposes problems in code that developers didn't foresee when writing the code. It's essential for this stage to produce feedback to developers quickly, while the problem space is still fresh in their minds and they can maintain the state of flow.

Deploy stages

Once we have a built a runnable instance of our code that has passed all predefined tests, we're ready to deploy it. There are usually multiple deploy environments, for example a "beta" or "staging" environment which is used internally by the product team, and a "production" environment for end users.

Teams that have embraced the Agile model of development—which is guided by tests and real-time monitoring—usually deploy work-in-progress manually to a staging environment for additional manual testing and review, and automatically deploy approved changes from the master branch to production.

Examples of CI/CD pipelines

A pipeline can start very simple. Here's an example of a pipeline for a Go project which compiles the code, checks code style and runs automated tests in two parallel jobs:

Golang CI pipeline

The pipeline is implemented with Semaphore, a cloud-based CI/CD service.

Here's a more complex example of a pipeline that builds, tests and deploys a microservice to a Kubernetes cluster:

Kubernetes CI/CD pipeline

And here's how to build it:

Semaphore documentation provides more examples of CI/CD pipelines.

Additional benefits of pipelines

Having a CI/CD pipeline has more positive effects than simply making what was previously done a little bit more efficient:

  • Developers can stay focused on writing code and monitoring the behavior of the system in production.
  • QA and product stakeholders have easy access to the latest, or any, version of the system.
  • Product updates are not stressful.
  • Logs of all code changes, test and deployments are available for inspection at any time.
  • Rolling back to a previous version in the event of a problem is a routine push-button action.
  • A fast feedback loop helps build an organizational culture of learning and responsibility.

Happy building!

Posted on by:

markoa profile

Marko Anastasov

@markoa

Let's learn together. 🎄 Semaphore CI/CD cofounder. @markoa on Twitter.

Discussion

markdown guide
 

Just a price comparsion for others interested.
CodeBuild is really in-expensive.

The entire CI feedback loop should run in less than 10 minutes.

Semaphore

e1-standard-2 - 4 GB memory, 2 vCPU
Semaphore is $0.00025/sec

0.00025*600=0.15

So thats 15¢ per 10 minutes

1,300 minutes free every month

AWS CodeBuild

build.general1.small - 3 GB memory, 2 vCPU
AWS CodeBuild costs $0.005/min

So thats 0.5¢ per 10 minutes

100 minutes free every month

 

Thanks for bringing this up. The comparison is valid if both services are able to do the same amount of work in 10 minutes. :) I’ll share a performance benchmark soon to shed more light on this.

Another major difference is that Semaphore prices are all inclusive. AWS charges you separately for log transfer and storage, network, other artifacts storage, etc.

 

GCP Cloud Build

n1-standard-1 - 3.75GB memory, 1vCPU
GCP Cloud Build costs $0.003/min

So thats 0.03¢ per 10 minutes

First 120 minutes per day are free

Reference

gcp

 

Where's the Azure person with the prices here?

 

Or roll your own with screwdriver.cd/
and pay what need to. Hint, run it right on your workstation in a container

 

This is an excellent article with valuable information for beginners. It's even more awesome with all the links to other articles peppered in the article. I'll read them later, thank you

 

This is one of the most flexible and relatable explanation on ci/CD have seen.