How to Build a CI/CD Pipeline from Scratch

Sherdil Cloud — Thu, 11 Jun 2026 14:07:41 +0000

TL;DR: Teams with mature CI/CD pipelines deploy 208× more frequently, experience 60% fewer deployment failures, and recover 96× faster (DORA State of DevOps Report). A production-ready pipeline builds in five stages: source control with branch protection → three-layer automated testing → containerized builds with vulnerability scanning → multi-environment deployment with blue-green/canary strategies → post-deploy monitoring with automated rollback. Most teams ship a basic pipeline in 1–2 weeks and a production-ready one in 4–8 weeks.

Building a CI/CD pipeline from scratch is one of the highest-leverage investments an engineering team can make. A well-designed pipeline transforms deployment from a manual, error-prone process that takes hours into an automated, reliable workflow that completes in minutes.

At Sherdil Cloud, we've built CI/CD pipelines for organizations across Pakistan, the UAE, and the United States since 2014 — for Python monoliths, Node.js microservices, containerized Java enterprise apps, and serverless functions. The principles of effective CI/CD stay consistent regardless of stack.

What is a CI/CD pipeline?

A CI/CD pipeline is an automated workflow that takes code from a developer's commit through testing, building, and deployment stages without manual intervention. CI and CD are related but distinct:

Continuous Integration (CI) — Every developer merges code into the shared repo at least daily. Each merge triggers automated builds and tests, catching integration problems early when they're cheap to fix.
Continuous Delivery (CD) — Every successful build auto-deploys to staging and is available for one-click production deployment. Production still requires a human decision.
Continuous Deployment (CD) — Every commit that passes tests deploys to production automatically. Safer than manual deployment because every change is small, tested, and easily reversible.

This guide builds toward Continuous Delivery, with the option to enable Continuous Deployment once your test suite and monitoring provide enough confidence.

Source control setup

Every CI/CD pipeline starts with source control. If your team isn't using Git with a structured branching strategy, fix that before anything else.

Choose a Git platform. GitHub, GitLab, or Bitbucket. All three provide CI/CD capabilities. GitHub Actions and GitLab CI are the most popular and best-documented.

Establish a branching strategy. For most teams, trunk-based development with feature branches works best:

The main branch always reflects deployable code
Developers create short-lived feature branches for each task
Feature branches merge to main through pull requests requiring at least one review
The CI pipeline runs on every pull request and every merge to main

Protect the main branch:

Require pull request reviews before merging
Require the CI pipeline to pass before merging
Prevent direct pushes (all changes go through pull requests)
Enable automatic branch deletion after merge

Automated testing

Automated tests are the backbone of any pipeline. Without reliable tests, automated deployment is just automated risk. Structure your suite in three layers — the canonical test pyramid: many fast unit tests at the base, fewer integration tests in the middle, a small set of end-to-end tests at the top.

Test layer	What it verifies	Tools	Run frequency	Time budget	Coverage target
Unit tests	Individual functions/methods in isolation	Jest, PyTest, JUnit, RSpec	Every PR + every commit	<5 min full suite	70–80% on business logic
Integration tests	Components working together (DB, API, service-to-service)	TestContainers, Supertest, Postman	Every merge to main	5–15 min	Cover critical paths
End-to-end tests	Critical user flows in a real browser	Cypress, Playwright, Selenium	Before production deploy	15–30 min	5–10 critical journeys

Coverage advice: Aim for 70–80% code coverage on business logic, not 100% everywhere. Chasing 100% wastes effort on trivial code (getters, constructors) and creates fragile tests that break on every refactor.

Build and artifact creation

After tests pass, the pipeline builds your application and creates deployable artifacts.

Containerized applications. Write a Dockerfile that installs dependencies, copies application code, and defines the startup command. Tag images with the Git commit hash (not :latest) for traceability. Push to a container registry: Amazon ECR, Google Artifact Registry, Azure Container Registry, or Docker Hub.

Non-containerized applications. The build stage compiles code, bundles assets, and packages the app into a deployable format: a JAR for Java, a wheel for Python, a zip archive for serverless functions.

Speed up builds with caching — reduce repeat build time by 50–80%:

Docker layer caching avoids rebuilding unchanged layers
Dependency caching (Maven .m2, Node node_modules, pip wheels) avoids re-downloading unchanged packages
Build artifact caching in the CI platform avoids recompiling unchanged modules

Sign and scan artifacts before deployment. Container image scanning with Trivy, Snyk, or Grype identifies known vulnerabilities in base images and dependencies. Fail the pipeline if critical or high-severity vulnerabilities are detected.

Deployment stages

A production-ready pipeline deploys through multiple environments, each adding validation before reaching users.

Environment	Receives	Purpose	Tests run
Development	Every successful build from feature branches	Devs test changes in a complete environment before merging	Smoke tests
Staging	Every successful build from main	Final validation gate; mirrors production in config, infra, data	Integration + end-to-end
Production	After staging validation passes	Real user traffic	Health checks + monitoring

Choosing a deployment strategy — pick based on the app's failure tolerance and your monitoring maturity:

Strategy	How it works	Best for	Rollback speed	Complexity
Blue-green	Two identical prod environments; new version deploys to the inactive one; traffic switches all at once	Stateless apps with budget for double infra	Seconds	Medium
Rolling	Gradually replaces old instances with new ones; pauses on health-check failure	Most workloads; default for Kubernetes Deployments	Minutes	Low
Canary	Routes 5–10% of traffic to the new version; monitors metrics; gradually increases	High-traffic apps where small errors must be caught fast	Seconds	High

Monitoring and rollback

Deployment isn't the final step. Monitoring and automated rollback complete the pipeline.

Track deployment health for 15–30 minutes. After each deployment, monitor error rates, response latency (p95 / p99), and throughput. Compare against pre-deployment baselines. If error rates exceed a threshold (we recommend 2× the baseline error rate), trigger an automatic rollback.

Notify the team of every deployment. Use Slack, Microsoft Teams, or email to broadcast what was deployed, to which environment, by whom, and the outcome (success, failure, rollback).

Maintain a deployment history. Record every production deployment with version, timestamp, deployer, and outcome. The first question after a production issue is always "what changed recently?"

Automate rollback. Configure automated rollback that reverts to the previous known-good version when monitoring detects problems. Manual rollback under pressure is error-prone; automated rollback is consistent.

A real engagement: UAE fintech CI/CD migration

In a 2024 engagement with a UAE-based fintech client (10 microservices, 14-engineer team, manual deploys via shell scripts), we built a GitHub Actions pipeline over 5 weeks.

Metric	Before pipeline	After 5 weeks
Deployment frequency	1 per week	18 per week
Average build time	22 minutes	4 minutes
Build failure recovery	90 min (manual)	<5 min (auto-rollback)
Deployment-tied incidents	6 per quarter	1 per quarter
Engineer deploy hours / week	~9 hours	~1 hour
First-time deploy success rate	~75%	~98%

Pipeline stack: GitHub Actions for orchestration. Jest and PyTest for unit testing. Cypress for end-to-end. Trivy for image scanning. Amazon ECR for the registry. EKS with rolling deployments for runtime. Auto-rollback triggered by Datadog watchdog alerts when post-deploy error rates exceeded 2× baseline.

The kicker: The fintech closed its Series B five months after the engagement. Technical due diligence specifically cited the deployment-frequency increase (1/wk → 18/wk) as evidence of engineering discipline.

Choosing CI/CD tools

Tool	Best for	Free tier	Hosting model
GitHub Actions	Teams already on GitHub; broad marketplace	2,000 min/month for private repos	Hosted
GitLab CI	All-in-one DevOps platform	400 min/month on free tier	Hosted or self-managed
Jenkins	Enterprises needing maximum customization	Open-source; pay only for infra	Self-managed
AWS CodePipeline	AWS-centric infra; tight IAM integration	Pay per active pipeline	Hosted
Azure DevOps Pipelines	Azure / Microsoft stack workflows	1,800 min/month free for public	Hosted or self-managed
Google Cloud Build	GCP-centric / container-first workflows	120 build-min/day free	Hosted

The best tool is the one your team will actually use consistently. Choose based on your existing workflow, not feature comparisons.

Frequently asked questions

What is a CI/CD pipeline and why is it important?
An automated workflow that takes code from commit through testing, building, and deployment. It eliminates manual deployment errors, enables faster release cycles, catches bugs early, and provides a repeatable, auditable process. Mature pipelines see 60% fewer deployment failures and recover 96× faster (DORA).

How long does it take to build a CI/CD pipeline from scratch?
A basic pipeline with automated testing and staging deployment: 1–2 weeks for a simple app. A production-ready pipeline with multi-stage deploys, security scanning, blue-green/canary strategies, monitoring, and automated rollback: typically 4–8 weeks.

Which CI/CD tool should I use: GitHub Actions, GitLab CI, or Jenkins?
On GitHub? Start with GitHub Actions. Want an all-in-one platform? GitLab CI. Need maximum customization with self-hosted ops? Jenkins. Single-cloud workloads? Consider the provider's native tool (AWS CodePipeline, Azure DevOps).

What tests should run in a CI/CD pipeline?
Three layers: unit tests (fast, business logic, every commit), integration tests (component interactions, every merge to main), and focused end-to-end tests (5–10 critical journeys, before production). Plus static analysis, dependency scanning, and container image scanning if you deploy containers.

Can CI/CD work for small teams?
Yes — small teams benefit most. A 2-person team spending 4 hours/week on manual deployments saves 200+ hours per year by automating. Tools like GitHub Actions make setup accessible regardless of DevOps experience.

This is a step-by-step companion to our longer guide to building CI/CD pipelines. Originally published on the Sherdil Cloud blog.

DEV Community: Sherdil Cloud