DEV Community

IKER ALBERTO SIERRA RUIZ
IKER ALBERTO SIERRA RUIZ

Posted on • Edited on

CI/CD and Automated Testing: A Real-World Comparison of 8 Tools with Code Examples

#automation #cicd #devops #testing

Choosing a CI/CD tool to automate your testing pipelines shouldn't be a decision based purely on marketing. In this post we compare 8 widely used tools in the industry — GitLab Pipelines, GitHub Actions, Jenkins, CircleCI, TeamCity, Travis CI, Bitbucket Pipelines, Tekton, and Harness — showing real configurations that run a typical testing pipeline: installing dependencies, running the linter, running unit tests, and publishing a coverage report.

At the end you'll find a public reference repository you can clone, adapt, and experiment with.

What we'll compare

For a fair comparison, every example solves the same scenario:

  1. Checkout the code
  2. Install dependencies (Node.js as an example)
  3. Linter (eslint)
  4. Unit tests (jest) with a coverage report

5. Publish the report/artifact

1. GitHub Actions

Model: GitHub-native SaaS, declarative YAML, hosted or self-hosted runners.

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - run: npm ci
      - run: npm run lint
      - run: npm test -- --coverage

      - uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: coverage/
Enter fullscreen mode Exit fullscreen mode

Strengths: native PR integration, huge marketplace of reusable actions, very gentle learning curve.
Limitations: limited free minutes on private repos; the third-party actions ecosystem requires checking the trust chain (supply chain).

Official public reference repo: actions/starter-workflows

2. GitLab CI/CD (Pipelines)

Model: built into GitLab, self-hosted or SaaS, based on stages and jobs.

# .gitlab-ci.yml
stages:
  - install
  - lint
  - test

variables:
  NODE_ENV: "test"

install_deps:
  stage: install
  image: node:20
  script:
    - npm ci
  cache:
    paths:
      - node_modules/

lint:
  stage: lint
  image: node:20
  script:
    - npm run lint

unit_tests:
  stage: test
  image: node:20
  script:
    - npm test -- --coverage
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    paths:
      - coverage/
Enter fullscreen mode Exit fullscreen mode

Strengths: multi-project pipelines (include), native coverage regex visible right in the MR, very powerful if you already use GitLab as a full platform (issues, registry, etc.).
Limitations: shared SaaS runner performance can lag behind GitHub Actions during usage spikes.

3. Jenkins

Model: self-hosted, extensible via plugins, Pipeline as Code with a Jenkinsfile (Groovy DSL).

// Jenkinsfile
pipeline {
    agent { docker { image 'node:20' } }

    stages {
        stage('Install') {
            steps {
                sh 'npm ci'
            }
        }
        stage('Lint') {
            steps {
                sh 'npm run lint'
            }
        }
        stage('Test') {
            steps {
                sh 'npm test -- --coverage'
            }
        }
    }

    post {
        always {
            archiveArtifacts artifacts: 'coverage/**', fingerprint: true
            junit 'reports/junit/*.xml'
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Strengths: full control over the environment, thousands of plugins, ideal for on-premise infrastructure or strict compliance requirements.
Limitations: you own the maintenance (patching, scaling agents), steeper learning curve, the Groovy syntax can be less intuitive.

4. CircleCI

Model: SaaS or self-hosted (server), YAML configuration with reusable orbs.

# .circleci/config.yml
version: 2.1

orbs:
  node: circleci/node@5.2.0

jobs:
  test:
    docker:
      - image: cimg/node:20.11
    steps:
      - checkout
      - node/install-packages:
          pkg-manager: npm
      - run: npm run lint
      - run: npm test -- --coverage
      - store_artifacts:
          path: coverage

workflows:
  ci:
    jobs:
      - test
Enter fullscreen mode Exit fullscreen mode

Strengths: orbs drastically reduce boilerplate, good test parallelization support and smart caching.
Limitations: credit-based pricing can be confusing when estimating costs at scale.

5. TeamCity

Model: JetBrains, self-hosted with a cloud option, configured via UI or versioned Kotlin DSL.

// .teamcity/settings.kts (excerpt)
import jetbrains.buildServer.configs.kotlin.*
import jetbrains.buildServer.configs.kotlin.buildSteps.script

project {
    buildType(CiPipeline)
}

object CiPipeline : BuildType({
    name = "CI Pipeline"

    steps {
        script {
            name = "Install"
            scriptContent = "npm ci"
        }
        script {
            name = "Lint"
            scriptContent = "npm run lint"
        }
        script {
            name = "Test"
            scriptContent = "npm test -- --coverage"
        }
    }

    artifactRules = "coverage => coverage.zip"
})
Enter fullscreen mode Exit fullscreen mode

Strengths: excellent for corporate .NET/Java environments, testing composite builds ("build chains"), good historical traceability.
Limitations: per-agent licensing can get expensive for large teams; less traction in the open-source community compared to GitHub Actions or GitLab CI.

6. Travis CI

Model: one of the pioneers of CI in the cloud, simple configuration based on .travis.yml.

# .travis.yml
language: node_js
node_js:
  - "20"

cache: npm

install:
  - npm ci

script:
  - npm run lint
  - npm test -- --coverage

after_success:
  - bash <(curl -s https://codecov.io/bash)
Enter fullscreen mode Exit fullscreen mode

Strengths: very readable configuration, ideal for small/medium open-source projects.
Limitations: after the 2020-2021 business model change it lost ground to GitHub Actions; the free plan for open source is more limited than it used to be.

7. Bitbucket Pipelines

Model: built into Bitbucket Cloud, YAML with reusable pipes.

# bitbucket-pipelines.yml
image: node:20

pipelines:
  default:
    - step:
        name: Install & Test
        caches:
          - node
        script:
          - npm ci
          - npm run lint
          - npm test -- --coverage
        artifacts:
          - coverage/**
Enter fullscreen mode Exit fullscreen mode

Strengths: minimal configuration for teams already in the Atlassian ecosystem (Jira, Bitbucket), good support for per-environment "deployments."
Limitations: the pipes marketplace is smaller than GitHub Actions'.

8. Tekton

Model: cloud-native CI/CD on Kubernetes, based on CRDs (Task, Pipeline, PipelineRun). It's not SaaS: it runs inside your cluster.

# tekton/task-test.yaml
apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: run-tests
spec:
  workspaces:
    - name: source
  steps:
    - name: install
      image: node:20
      workingDir: $(workspaces.source.path)
      script: npm ci

    - name: lint
      image: node:20
      workingDir: $(workspaces.source.path)
      script: npm run lint

    - name: test
      image: node:20
      workingDir: $(workspaces.source.path)
      script: npm test -- --coverage
Enter fullscreen mode Exit fullscreen mode

Strengths: maximum portability (runs the same on any Kubernetes), extensible with custom CRDs, a good choice if you already have an internal developer platform (IDP).
Limitations: requires operating and maintaining Kubernetes; the steepest learning curve on this list.

9. Harness

Model: commercial CI/CD platform focused on visual pipelines + YAML, strong in CD and feature flags.

# harness/ci-pipeline.yaml (excerpt)
pipeline:
  name: ci-pipeline
  stages:
    - stage:
        name: Build and Test
        type: CI
        spec:
          execution:
            steps:
              - step:
                  type: Run
                  name: Install
                  spec:
                    command: npm ci
              - step:
                  type: Run
                  name: Lint
                  spec:
                    command: npm run lint
              - step:
                  type: Run
                  name: Test
                  spec:
                    command: npm test -- --coverage
Enter fullscreen mode Exit fullscreen mode

Strengths: good visual UX for teams that prefer not to write YAML from scratch, advanced CD capabilities (canary, blue/green) built in.
Limitations: commercial product, the free tier is more limited than purely open-source alternatives.

Quick comparison table

Tool Hosting Configuration Learning curve Best for
GitHub Actions SaaS / self-hosted YAML Low Projects already on GitHub
GitLab CI SaaS / self-hosted YAML Low-Medium Teams on GitLab, monorepos
Jenkins Self-hosted Groovy DSL High Compliance, full control
CircleCI SaaS / self-hosted YAML + Orbs Low Fast parallelization
TeamCity Self-hosted / cloud UI or Kotlin DSL Medium-High Corporate .NET/Java environments
Travis CI SaaS YAML Low Small OSS projects
Bitbucket Pipelines SaaS YAML Low Atlassian ecosystem
Tekton Kubernetes (self-hosted) CRDs YAML High Cloud-native platforms
Harness Commercial SaaS YAML/visual UI Medium Advanced CD, feature flags

Public example repository

To see these pipelines actually working (not just in a blog post), you can explore and clone the official GitHub Actions templates repository:

🔗 https://github.com/actions/starter-workflows

If you want to compare several tools on the same source code, the practical recommendation is:

  1. Fork any small Node.js repo (for example, a sample REST API).
  2. Add the configuration files shown above in parallel (.github/workflows/ci.yml, .gitlab-ci.yml, Jenkinsfile, .circleci/config.yml, etc.).
  3. Connect the repo to each platform (they all offer a free tier for testing) and compare build times, log readability, and ease of debugging. That "one repo, multiple pipelines" exercise is the most honest way to decide which one fits your team best, since you'll see the real differences in UX, speed, and reporting — not just syntax.

Conclusion

There's no "best" tool in the abstract:

  • If you already live in GitHub, GitHub Actions is the path of least friction.
  • If you need full control or must meet strict compliance requirements, Jenkins is still very relevant.
  • If you run on Kubernetes and want something truly cloud-native, Tekton is the most aligned option.
  • If your priority is advanced CD (canary, feature flags) with a visual layer, Harness delivers value quickly.

Top comments (1)

Collapse
 
ji_ll_4858c7d4b87a30 profile image
Jimmy Llica •

¡Qué gran aporte, Iker! Muy interesante ver las configuraciones de los pipelines lado a lado en lugar de solo leer comparativas teóricas. Ayuda muchísimo a visualizar la fricción y la curva de aprendizaje de cada herramienta, especialmente el contraste entre los YAML de GitHub/GitLab y los CRDs de Tekton. ¡Excelente recomendación la de armar el repositorio de prueba para medir tiempos reales!