DEV Community

Pavan Kumar Sadashiv
Pavan Kumar Sadashiv

Posted on

Testing Across Everything: Mastering Matrix Builds & Parallelism in GitLab CI/CD

#gitlab #cicd #devops #testing

Testing Across Everything: Mastering Matrix Builds & Parallelism in GitLab CI/CD

When maintaining an open-source library, cross-platform CLI tool, or multi-tenant backend, you can never assume that code passing on Node.js 22 or Python 3.12 on Linux will run without errors on Node 18 or Python 3.10 on macOS.

Historically, testing multiple versions in CI meant one of two terrible options:

  1. The Sequential Slow-Down: Running tests for Python 3.10, then 3.11, then 3.12 one after another inside a single job. A 5-minute test suite turns into a 25-minute crawl.
  2. The Copy-Paste YAML Nightmare: Duplicating 40 lines of CI job configuration 6 times with different image tags.

Fortunately, GitLab CI/CD features native Parallel Matrix Builds (parallel: matrix:).

In this practical guide, I will show you how to write clean, concise matrix configurations that fan out across dozens of language versions, database engines, and CPU architectures in parallel.

1. How Parallel Matrix Builds Work 

+-------------------------------------------------------------+
|               Single Job Definition in YAML                 |
|                                                             |
|   parallel:                                                 |
|     matrix:                                                 |
|       - RUNTIME: [3.10, 3.11, 3.12]                         |
|         DATABASE: [postgres, mysql]                         |
+-------------------------------------------------------------+
                              |
                              v (GitLab fans out 6 parallel jobs)
+-------------------------------------------------------------+
|  [Job 1] Python 3.10 + Postgres (Runs on Runner A)         |
|  [Job 2] Python 3.10 + MySQL    (Runs on Runner B)         |
|  [Job 3] Python 3.11 + Postgres (Runs on Runner C)         |
|  [Job 4] Python 3.11 + MySQL    (Runs on Runner D)         |
|  [Job 5] Python 3.12 + Postgres (Runs on Runner E)         |
|  [Job 6] Python 3.12 + MySQL    (Runs on Runner F)         |
|                                                             |
|  * All 6 jobs execute simultaneously! Total time: ~3 mins!  |
+-------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

2. Basic Matrix: Testing Multi-Version Runtimes

Here is how to test a Node.js library across LTS releases (18-alpine, 20-alpine, 22-alpine) in parallel:

stages:
  - test

test_node_matrix:
  stage: test
  image: node:$NODE_VERSION
  parallel:
    matrix:
      - NODE_VERSION:
          - "18-alpine"
          - "20-alpine"
          - "22-alpine"
  script:
    - node -v
    - npm ci
    - npm test
Enter fullscreen mode Exit fullscreen mode

What You See in GitLab:

In the pipeline graph, GitLab automatically labels the dynamically created jobs:

  • test_node_matrix: [18-alpine]
  • test_node_matrix: [20-alpine]
  • test_node_matrix: [22-alpine]

Each job runs concurrently on available runners, cutting test feedback latency by 66%.

3. Multi-Dimensional Matrix: Runtimes × Databases

When your backend supports multiple database backends, combine multiple variables into a 2D matrix:

test_backend_grid:
  stage: test
  image: python:$PYTHON_VERSION-slim
  services:
    - name: $DB_IMAGE
      alias: db
  parallel:
    matrix:
      - PYTHON_VERSION: ["3.11", "3.12"]
        DB_IMAGE: ["postgres:16-alpine", "mariadb:11.4"]
  variables:
    POSTGRES_DB: test_db
    POSTGRES_PASSWORD: secretpassword
    MARIADB_DATABASE: test_db
    MARIADB_ROOT_PASSWORD: secretpassword
  script:
    - python -V
    - pip install -r requirements.txt
    - pytest tests/
Enter fullscreen mode Exit fullscreen mode

GitLab computes the Cartesian product ($2 \times 2 = 4$ jobs), provisioning isolated ephemeral database service containers for each permutation.

4. Complex Permutations & Specific Exclusions

What if a specific runtime combination is unsupported (e.g., Python 3.10 is incompatible with a modern library requiring Python 3.11+)?

You can define multiple independent matrix arrays within the same block:

test_compatibility:
  stage: test
  image: $BASE_IMAGE
  parallel:
    matrix:
      # Block 1: Linux x86_64 across full stack
      - BASE_IMAGE: ["ubuntu:24.04", "debian:12-slim"]
        ARCH: ["amd64"]
      # Block 2: ARM64 targets (scoped only to Alpine)
      - BASE_IMAGE: ["alpine:3.20"]
        ARCH: ["arm64"]
  script:
    - echo "Compiling binary for $BASE_IMAGE on $ARCH..."
Enter fullscreen mode Exit fullscreen mode

5. Parallel Count vs. Matrix (Fast Test Sharding)

If you don't have multiple language versions, but your test suite has 2,000 slow unit tests, you can use integer test sharding (parallel: N):

shard_tests:
  stage: test
  image: ruby:3.2
  parallel: 4 # Spawns 4 parallel runners: 1/4, 2/4, 3/4, 4/4
  script:
    # Use knapsack or rspec test sharding
    - bundle exec rspec $(bin/split_specs --index $CI_NODE_INDEX --total $CI_NODE_TOTAL)
Enter fullscreen mode Exit fullscreen mode

$CI_NODE_INDEX (e.g. 1) and $CI_NODE_TOTAL (e.g. 4) are injected automatically by GitLab, allowing your test runner to slice the test directory evenly across runners.

6. Key Takeaways

  1. Eliminate Duplicated YAML: Let GitLab generate Cartesian product permutations dynamically using parallel: matrix:.
  2. True Concurrent Feedback: Multi-version validation finishes in the time of a single run.
  3. Combine with Caching: Ensure dependency caches are partitioned by runtime version (key: "$CI_JOB_NAME") to prevent cross-version cache poisoning.

Resources & Links

How many permutations does your test suite run across? Share your matrix optimization tricks in the comments below!

Top comments (0)