The Quest Begins (The “Why”)
Honestly, I used to dread Monday mornings because my CI pipeline felt like a broken cart wheel—wobbling, squeaky, and always threatening to throw me off the road. I’d push a commit, wait fifteen minutes for a build that never finished, then scroll through logs that looked like ancient runes. The worst part? When it finally passed locally, the same code would explode on the staging server because some environment variable was missing or a Docker image was built with the wrong tag. I felt like Frodo carrying the One Ring through Mordor—every step was a struggle, and I kept wondering if there was a smoother path to the mountain.
That frustration sparked a question: What does a CI/CD pipeline that actually works look like? I wanted something that gave me fast feedback, caught bugs before they escaped, and let me ship with confidence—no more guessing games.
The Revelation (The Insight)
The breakthrough came when I stopped treating the pipeline as a monolithic script and started seeing it as a series of small, testable spells. Each spell (job) does one thing well: lint, test, build, scan, deploy. When you chain them with clear artifacts and cache dependencies, the whole thing becomes predictable.
I also learned that the real magic isn’t in the tool—it’s in the contract you define between stages. If the build job outputs a Docker image tagged with the Git SHA, the deploy job can trust that artifact without re‑building. If the test job fails, the pipeline stops early, saving you minutes (or hours) of wasted compute.
Finally, I embraced the idea of “pipeline as code”. Storing the workflow in the same repo as the application means every change to the pipeline is reviewed, versioned, and can be rolled back just like any other feature.
Wielding the Power (Code & Examples)
Below are three flavors of the same pipeline—GitHub Actions, GitLab CI, and Jenkins—showing the before (a fragile, all‑in‑one script) and the after (a clean, modular setup).
GitHub Actions
Before – a single, long job
name: CI
on: [push, pull_request]
jobs:
build-and-deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Node
uses: actions/setup-node@v3
with:
node-version: '20'
- run: npm ci
- run: npm run lint # fails silently sometimes
- run: npm test # runs unit + integration together
- run: npm run build # creates dist/
- name: Docker build
run: |
docker build -t myapp:${{ github.sha }} .
docker push myapp:${{ github.sha }}
- name: Deploy to staging
uses: appleboy/ssh-action@v0.1.8
with:
host: ${{ secrets.STAGING_HOST }}
username: ${{ secrets.STAGING_USER }}
key: ${{ secrets.STAGING_SSH_KEY }}
script: |
docker pull myapp:${{ github.sha }}
docker stop old || true
docker run -d --name myapp-${{ github.sha }} -p 80:80 myapp:${{ github.sha }}
The problem? If lint passes but tests fail, you’ve already built and pushed a Docker image. You also waste time re‑installing Node on every retry.
After – modular jobs with caching
name: CI/CD
on:
push:
branches: [ main ]
pull_request:
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: '20'
- run: npm ci
- run: npm run lint
test:
needs: lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: '20'
- run: npm ci
- run: npm test
build:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: '20'
- run: npm ci
- run: npm run build
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Log in to Docker Hub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v4
with:
context: .
push: true
tags: myapp/${{ github.sha }}
deploy-staging:
needs: build
runs-on: ubuntu-latest
steps:
- name: Deploy via SSH
uses: appleboy/ssh-action@v0.1.8
with:
host: ${{ secrets.STAGING_HOST }}
username: ${{ secrets.STAGING_USER }}
key: ${{ secrets.STAGING_SSH_KEY }}
script: |
docker pull myapp:${{ github.sha }}
docker stop old || true
docker run -d --name myapp-${{ github.sha }} -p 80:80 myapp/${{ github.sha }}
Now each stage only runs when its predecessor succeeds, Docker Buildx caches layers, and we never push an image that hasn’t passed lint + test.
GitLab CI
Before – a monstrous .gitlab-ci.yml
stages:
- build
- test
- deploy
build:
stage: build
script:
- npm ci
- npm run build
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
test:
stage: test
script:
- npm ci
- npm test
deploy:
stage: deploy
script:
- docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- docker run -d -p 80:80 $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
If the build job fails, the test job still pulls the old image from the registry, leading to confusion.
After – using needs and caching
stages:
- lint
- test
- build
- deploy
variables:
DOCKER_DRIVER: overlay2
cache:
key: ${CI_JOB_NAME}
paths:
- node_modules/
lint:
stage: lint
script:
- npm ci
- npm run lint
test:
stage: test
needs: ["lint"]
script:
- npm ci
- npm test
build:
stage: build
needs: ["test"]
script:
- npm ci
- npm run build
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
deploy:
stage: deploy
needs: ["build"]
script:
- docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
- docker run -d -p 80:80 $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
The needs keyword guarantees that deploy only runs after a successful build, and the cache saves node_modules between jobs, cutting minutes off each run.
Jenkins (Declarative Pipeline)
Before – a scripted pipeline with everything in one stage
pipeline {
agent any
stages {
stage('Build') {
steps {
sh 'npm ci'
sh 'npm run build'
sh 'docker build -t myapp:${env.GIT_COMMIT} .'
sh 'docker push myapp:${env.GIT_COMMIT}'
}
}
stage('Test') {
steps {
sh 'npm ci'
sh 'npm test'
}
}
stage('Deploy') {
steps {
sh 'docker pull myapp:${env.GIT_COMMIT}'
sh 'docker run -d -p 80:80 myapp:${env.GIT_COMMIT}'
}
}
}
}
If the Build stage fails, the Test stage still tries to run npm ci on a stale workspace, and the Deploy stage may pull an image that never got pushed.
After – parallelism, clean workspaces, and explicit artifacts
pipeline {
agent any
options {
timeout(time: 30, unit: 'MINUTES')
timestamps()
}
stages {
stage('Lint') {
steps {
checkout scm
sh 'npm ci'
sh 'npm run lint'
}
}
stage('Test') {
steps {
checkout scm
sh 'npm ci'
sh 'npm test'
}
}
stage('Build') {
steps {
checkout scm
sh 'npm ci'
sh 'npm run build'
sh 'docker build -t myapp:${env.GIT_COMMIT} .'
sh 'docker push myapp:${env.GIT_COMMIT}'
}
}
stage('Deploy') {
steps {
script {
// Only run on main branch
if (env.BRANCH_NAME == 'main') {
sh 'docker pull myapp:${env.GIT_COMMIT}'
sh 'docker run -d -p 80:80 myapp:${env.GIT_COMMIT}'
}
}
}
}
}
post {
always {
cleanWs()
}
success {
echo '🚀 Pipeline succeeded!'
}
failure {
echo '💥 Something went wrong – check the logs.'
}
}
}
Now each stage gets a fresh checkout, we fail fast, and we only deploy to production (or staging) when the branch is main. The post block guarantees a clean workspace and gives us instant feedback.
Why This New Power Matters
With these patterns, I went from “I hope it works” to “I know it works.” The feedback loop dropped from fifteen minutes to under three, and I could ship multiple times a day without fear. My team started catching dependency bugs in the lint stage, security vulnerabilities in the build stage, and configuration drift only in the deploy stage—each isolated, each actionable.
Most importantly, the pipeline became documentation. New hires could look at the YAML/Jenkinsfile and understand the entire delivery process in a glance, which saved hours of onboarding meetings.
Your Turn
Pick one of the three examples above, clone a small project, and replace its CI file with the modular version. Run a commit, watch the stages fly by, and notice how fast you get a green check. Then try breaking one step on purpose—see how the pipeline stops early and saves you time.
What’s the first improvement you’ll make to your own pipeline? Share your before/after stories in the comments; I love hearing how fellow developers slay their CI dragons! 🚀
Top comments (0)