DEV Community

DAYAN ELVIS JAHUIRA PILCO
DAYAN ELVIS JAHUIRA PILCO

Posted on

CI/CD Testing Tools Compared: the Same Pipeline in GitHub Actions, GitLab CI and Jenkins

#cicd #devops #testing #tutorial

TL;DR: I compared the major CI/CD tools for test automation (GitHub Actions, GitLab CI, Jenkins, CircleCI, Travis CI, TeamCity, Bitbucket Pipelines, Tekton, Harness) — and instead of stopping at a table, I wrote the exact same test pipeline three times: once for GitHub Actions (running live), once for GitLab CI, and once as a Jenkinsfile, all in one public repo so you can compare the syntax line by line. Repo: github.com/Dayan-18/ci-tools-demo →

Why CI is where your tests actually live

A test suite that runs only on your laptop protects exactly one person: you. The moment tests run automatically on every push and pull request, they protect the whole team — and that's the job of a CI/CD tool. In my previous articles I used GitHub Actions to automate SAST scans, IaC scans and API tests. But GitHub Actions is one option among many. How do the alternatives compare?

The landscape

Tool Hosting Config Pricing (open source) Standout feature Watch out for
GitHub Actions SaaS (+ self-hosted runners) YAML in repo Free for public repos Marketplace with 20k+ reusable actions Vendor lock-in to GitHub
GitLab CI SaaS or self-hosted YAML in repo Free tier + minutes Single tool: repo, CI, registry, environments Complex YAML at scale
Jenkins Self-hosted Jenkinsfile (Groovy) 100% free, open source Total control, 1,900+ plugins You maintain the server, plugins break
CircleCI SaaS YAML Free tier Very fast, first-class Docker Credits model gets pricey
Travis CI SaaS YAML Limited free Historic pioneer for OSS Lost most mindshare after pricing changes
TeamCity Self-hosted or Cloud UI or Kotlin DSL Free tier Deep JetBrains IDE integration Heavier setup
Bitbucket Pipelines SaaS YAML Free tier Native Jira/Bitbucket integration Only useful inside Atlassian stack
Tekton Kubernetes CRDs (YAML) Free, CNCF Cloud-native building blocks Steep learning curve; it's a framework, not a product
Harness SaaS YAML/UI Free tier AI-assisted pipelines, deployment verification Enterprise-oriented complexity

Tables are nice, but syntax is what you live with every day. So let's compare the three most representative tools with real code: the same pipeline — install dependencies, run a pytest suite, publish a JUnit report — written three times.

The test suite (identical for all three)

A small shopping-cart calculator with 10 tests (happy paths, error paths, parametrized cases):

@pytest.mark.parametrize("amount,percent,expected", [
    (100.0, 0, 100.0),
    (100.0, 10, 90.0),
    (100.0, 100, 0.0),
    (59.99, 25, 44.99),
])
def test_apply_discount(amount, percent, expected):
    assert apply_discount(amount, percent) == expected
Enter fullscreen mode Exit fullscreen mode

Running locally:

test_calculator.py::test_subtotal_sums_price_times_quantity PASSED       [ 10%]
test_calculator.py::test_subtotal_empty_cart_is_zero PASSED              [ 20%]
test_calculator.py::test_subtotal_rejects_negative_values PASSED         [ 30%]
test_calculator.py::test_apply_discount[100.0-0-100.0] PASSED            [ 40%]
test_calculator.py::test_apply_discount[100.0-10-90.0] PASSED            [ 50%]
test_calculator.py::test_apply_discount[100.0-100-0.0] PASSED            [ 60%]
test_calculator.py::test_apply_discount[59.99-25-44.99] PASSED           [ 70%]
test_calculator.py::test_apply_discount_rejects_invalid_percent PASSED   [ 80%]
test_calculator.py::test_total_with_tax_default_18_percent PASSED        [ 90%]
test_calculator.py::test_total_with_tax_custom_rate PASSED               [100%]

============================== 10 passed ==============================
Enter fullscreen mode Exit fullscreen mode

Now, the same automation in three dialects.

Round 1: GitHub Actions (.github/workflows/tests.yml) 

name: Tests (GitHub Actions)

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.11", "3.12"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Run tests
        run: pytest test_calculator.py -v --junitxml=report-${{ matrix.python-version }}.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-report-${{ matrix.python-version }}
          path: report-${{ matrix.python-version }}.xml
Enter fullscreen mode Exit fullscreen mode

Notice the matrix: two Python versions tested in parallel with four extra lines. Reusable actions (actions/setup-python) replace what would be manual scripting elsewhere. This one is running live in the repo — check the Actions tab.

Round 2: GitLab CI (.gitlab-ci.yml) 

stages:
  - test

.test_template: &test_template
  stage: test
  before_script:
    - pip install -r requirements.txt
  script:
    - pytest test_calculator.py -v --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml

test:python3.11:
  <<: *test_template
  image: python:3.11

test:python3.12:
  <<: *test_template
  image: python:3.12
Enter fullscreen mode Exit fullscreen mode

Two things stand out. First, jobs declare a Docker image directly — the container is the environment, no setup action needed. Second, reports: junit is built into the platform: failed tests appear annotated in the merge request UI. The version matrix, though, requires YAML anchors (&template/<<:) — more manual than Actions' matrix.

Round 3: Jenkins (Jenkinsfile) 

pipeline {
    agent {
        docker { image 'python:3.12' }
    }

    stages {
        stage('Install') {
            steps {
                sh 'pip install -r requirements.txt'
            }
        }

        stage('Test') {
            steps {
                sh 'pytest test_calculator.py -v --junitxml=report.xml'
            }
        }
    }

    post {
        always {
            junit 'report.xml'
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Jenkins is the only one here that is not YAML — it's a Groovy DSL, which means real programming (variables, conditionals, shared libraries) when you need it. The post { always { junit ... } } block is Jenkins' classic test-report integration. The trade-off is operational: GitHub and GitLab run your pipeline on their servers; with Jenkins, you run the server, the agents, the plugins, and the upgrades.

What the comparison actually teaches

Reading the three files side by side (they're all in the repo), the pattern is obvious: every CI tool automates the same four verbs — trigger, environment, steps, report. The differences are dialect and philosophy: Actions optimizes for reusable building blocks, GitLab for platform integration, Jenkins for control and extensibility. The niche players follow the same verbs too: Tekton expresses them as Kubernetes resources, Bitbucket Pipelines as Atlassian-flavored YAML, Harness wraps them in an enterprise UI.

My recommendation for choosing: if your code lives on GitHub, use Actions; on GitLab, use GitLab CI — fighting your platform's native tool is rarely worth it. Choose Jenkins when you need on-premise control or heavy customization, and Tekton only if your team already breathes Kubernetes.

Conclusion

"Which CI tool should I learn?" is less important than understanding the shared model: trigger → environment → steps → report. Learn it once, and every tool becomes a syntax lookup. The full working example — one suite, three pipelines — is public here: github.com/Dayan-18/ci-tools-demo, with GitHub Actions running it live on every push.

Which CI tool does your team use — and would you switch? Tell me in the comments! 👇

Previous articles in this series: SAST with Bandit · IaC scanning with Checkov · API testing with pytest

References: GitHub Actions docs · GitLab CI docs · Jenkins Pipeline docs · Tekton · CircleCI

Top comments (0)