DEV Community

Timevolt
Timevolt

Posted on

May the CI/CD Be With You: Building Pipelines That Actually Work with GitHub Actions, GitLab, Jenkins

#devops #docker #kubernetes #cicd

The Quest Begins (The "Why")

Here's the thing: I used to stare at my CI dashboard like it was a Mordor lava flow—red, angry, and utterly hopeless. Every push felt like a gamble; unit tests would pass locally, then explode in the pipeline because a dependency version drifted or a secret was hard‑coded somewhere. I’d spend Friday nights SSH‑ing into a Jenkins slave, manually copying artifacts, and praying the deployment didn’t wipe the production database. The worst part? The feedback loop was so slow that by the time I figured out what broke, I’d already moved on to the next feature and forgotten the context. I knew there had to be a better way—something that felt less like slogging through swamp and more like gliding on a speeder bike through Endor’s forests.

The Revelation (The Insight)

The treasure I uncovered wasn’t a new tool; it was a shift in mindset. Treat your pipeline as first‑class code: version‑controlled, modular, and testable. Instead of monolithic scripts that try to do everything, break the workflow into small, reusable jobs that communicate through well‑defined artifacts. Cache dependencies, use matrix strategies for parallelism, and let the platform handle the heavy lifting (like Docker build‑kit or Kubernetes agents). When I started seeing each step as a spell in a wizard’s book—each with clear inputs, outputs, and side effects—the whole system became predictable. And the best part? The same principles work whether you’re using GitHub Actions, GitLab CI, or a good old Jenkinsfile.

Wielding the Power (Code & Examples)

Before: The Fragile GitHub Actions Workflow 

name: CI

on: [push, pull_request]

jobs:
  build-test-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2   # ❌ outdated, missing fetch-depth
      - name: Set up Node
        uses: actions/setup-node@v2
        with:
          node-version: '16'
      - run: npm ci
      - run: npm test
      - name: Deploy (hard‑coded secret!)
        env:
          API_KEY: ${{ secrets.API_KEY }}   # leaked if logs are public
        run: |
          curl -X POST https://api.example.com/deploy \
            -H "Authorization: Bearer $API_KEY"
Enter fullscreen mode Exit fullscreen mode

What’s wrong?

  • Checkout v2 doesn’t fetch tags, breaking version‑based releases.
  • No caching of node_modules → every run reinstalls from scratch.
  • The deploy step dumps the secret into the workflow logs if something goes wrong (yes, I’ve seen it).

After: A Robust, Reusable Workflow 

name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      artifact-id: ${{ steps.save-artifact.outputs.artifact-id }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # get all tags for versioning
      - name: Cache node_modules
        uses: actions/actions/cache@v3
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ hashFiles('package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-
      - run: npm ci
      - run: npm run build
      - name: Save build artifact
        id: save-artifact
        uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/
Enter fullscreen mode Exit fullscreen mode 
  test:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: dist
          path: ./dist
      - run: npm test
Enter fullscreen mode Exit fullscreen mode 
  deploy:
    needs: test
    runs-on: ubuntu-latest
    environment: production   # protects secrets, requires approval
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with:
          name: dist
          path: ./dist
      - name: Deploy to API
        env:
          API_KEY: ${{ secrets.API_KEY }}   # never appears in logs
        run: |
          curl -X POST https://api.example.com/deploy \
            -H "Authorization: Bearer $API_KEY" \
            -F "file=@./dist/bundle.js"
Enter fullscreen mode Exit fullscreen mode

What changed?

  • We pinned to actions/checkout@v4 for reliable fetching.
  • Added a caching step for ~/.npm; subsequent runs are lightning‑fast.
  • Split the pipeline into build → test → deploy with explicit needs dependencies, so failures stop early.
  • Used an environment to gate production deployment and keep secrets out of logs.
  • Artifacts are uploaded/downloaded instead of recomputing the build each job.

GitLab CI – From Monolith to Modular

Before

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/

test_job:
  stage: test
  script:
    - npm test
  dependencies:
    - build_job

deploy_job:
  stage: deploy
  script:
    - curl -X POST ...   # secret exposed in CI variables UI
Enter fullscreen mode Exit fullscreen mode

After

stages:
  - build
  - test
  - deploy

variables:
  NODE_OPTIONS: "--max_old_space_size=4096"

cache:
  key: "${CI_JOB_NAME}-${CI_COMMIT_REF_SLUG}-${CI_PIPELINE_ID}"
  paths:
    - node_modules/

.build_template: &build_template
  stage: build
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/

.test_template: &test_template
  stage: test
  script:
    - npm test
  needs: ["build_job"]

.deploy_template: &deploy_template
  stage: deploy
  script:
    - curl -X POST https://api.example.com/deploy \
          -H "Authorization: Bearer $API_KEY"
  environment:
    name: production
    url: https://example.com
  only:
    - main

build_job:
  <<: *build_template

test_job:
  <<: *test_template

deploy_job:
  <<: *deploy_template
Enter fullscreen mode Exit fullscreen mode

The secret API_KEY lives in GitLab’s protected variables, never shown in job logs. The cache section cuts install time from minutes to seconds.

Jenkinsfile – From Scripted Spaghetti to Declarative Clarity

Before (scripted, hard‑coded)

node {
    stage('Checkout') {
        checkout scm
    }
    stage('Build') {
        sh 'npm ci'
        sh 'npm run build'
        stash name: 'dist', includes: 'dist/**'
    }
    stage('Test') {
        unstash 'dist'
        sh 'npm test'
    }
    stage('Deploy') {
        unstash 'dist'
        sh '''curl -X POST https://api.example.com/deploy \
                -H "Authorization: Bearer $API_KEY" '''
    }
}
Enter fullscreen mode Exit fullscreen mode

After (declarative, with agents and options)

pipeline {
    agent any
    options {
        timeout(time: 20, unit: 'MINUTES')
        timestamps()
    }
    environment {
        API_KEY = credentials('api-key-prod')
    }
    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }
        stage('Build') {
            steps {
                sh 'npm ci'
                sh 'npm run build'
                stash name: 'dist', includes: 'dist/**'
            }
        }
        stage('Test') {
            steps {
                unstash 'dist'
                sh 'npm test'
            }
        }
        stage('Deploy') {
            steps {
                unstash 'dist'
                sh '''curl -X POST https://api.example.com/deploy \
                        -H "Authorization: Bearer ${API_KEY}" '''
            }
        }
    }
    post {
        always {
            cleanWs()
        }
        success {
            echo '🚀 Pipeline succeeded!'
        }
        failure {
            mail to: 'team@example.com',
                 subject: "Failed build: ${env.JOB_NAME} #${env.BUILD_NUMBER}",
                 body: "Check the console output: ${env.BUILD_URL}"
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Notice how we declare the agent, use credentials() binding (no secret echoing), and add a post section for notifications and cleanup. The pipeline is now self‑documenting and easier to extend.

Why This New Power Matters

When I switched to these patterns, the mean time to recovery dropped from hours to minutes. A flaky test? The pipeline fails fast, the blame is clear, and I can fix it before the next coffee break. Deployments feel like pressing a button on a lightsaber—confident, precise, and with a satisfying hum. Teams stopped blaming “the CI” and started owning their workflows because they could read them like a story. Plus, caching and parallelism shaved off roughly 40% of our build time, which meant more feature work and less waiting for green checks.

If you’re still wrestling with monolithic scripts, hard‑coded secrets, or caches that never warm up, give the modular approach a try. Start small: extract a single job into a reusable template, add a cache for your package manager, and protect your deployment step with an environment. You’ll be amazed how quickly the chaos turns into calm.

Your turn: Pick one pipeline you maintain, identify the step that repeats the most (think dependency install or artifact upload), and turn it into a cached, reusable piece. Share your before/after snippets in the comments—I’d love to see your quest’s loot! 🚀

Top comments (0)