🦊 GitLab CI: Automated Testing of Job Rules

• Initial thoughts
• 1. The problem: CI rules complexity
• 2. The solution: automated job list validation
  • gitlab-ci-local: the cornerstone
  • How it works
• 3. Setting up the test infrastructure
  • Directory structure
  • Test case definition
  • Reference CSV files
• 4. The validation script
• 5. The CI job
• 6. Workflow: adding or modifying jobs
• 7. Testing with rules:changes
  • The problem with rules:changes and gitlab-ci-local
  • The force-build label workaround
  • Per-module test cases
• 8. Documentation that writes itself
• Wrapping up
• Further reading

Initial thoughts

As a CICD engineer, you've likely experienced this frustrating scenario: you modify a job's rules: to optimize pipeline execution, only to discover later that some jobs no longer trigger in specific situations. Or worse, jobs that should be mutually exclusive now run together, wasting resources and causing confusion.

GitLab CI's rules: syntax is powerful but complex. With workflow:rules, job-level rules:, extends:, !reference, and changes: all interacting, predicting which jobs will run for a given pipeline type becomes increasingly difficult as your CI configuration grows.

What if we could automatically test that the right jobs appear for each pipeline type? This article presents a practical solution: using gitlab-ci-local to validate job presence across all your pipeline variants, with reference files that serve as both tests and documentation.

1. The problem: CI rules complexity

In a mature GitLab CI setup, you typically have multiple pipeline types — especially in mono-repos with complex workflow:rules and module-based builds. Here, we focus on testing that your rules produce the expected jobs.

Type	Trigger	Example
A	Merge Request	Developer pushes to a feature branch with an open MR
B	Protected Branch	Push to main, develop, or release branches
C	Manual Pipeline	Triggered from GitLab UI with variables
D	Tag (auto)	Tag push triggering preprod deployment
E	Tag (manual)	Tag pipeline for production deployment
F	Scheduled	Nightly builds, cache warmup, full test suites

Each pipeline type should have a specific set of jobs. For example:

• MR pipelines should run tests related to changed modules only
• Protected branch pipelines should run all tests and prepare deployable artifacts
• Scheduled pipelines might run expensive security scans or full regression tests

When you have 50+ jobs with complex rules:, maintaining this becomes a nightmare. Change one rule, break three jobs, sometimes in further pipeline types — the butterfly effect, CI edition. And if you've ever endured the slow feedback loop of CI YAML modifications, you know that discovering these regressions through push-and-pray is not sustainable.

2. The solution: automated job list validation

The solution is surprisingly simple and will be detailed step by step in this article:

1. Define test cases as variable files simulating each pipeline type
2. Use gitlab-ci-local --list-csv to get the jobs that would run
3. Compare with reference CSV files committed to the repository
4. Fail the CI if the job list differs from the expected reference

gitlab-ci-local: the cornerstone

gitlab-ci-local is an open-source tool that parses GitLab CI YAML locally. Among its many features, --list-csv outputs the jobs that would run given a set of variables, without actually executing anything.

This is perfect for our use case: we simulate pipeline conditions and check the resulting job list. Think of it as a flight simulator for your CI — all the turbulence, none of the crashes.

# Install globally
npm install -g gitlab-ci-local

# List jobs as CSV with custom variables
gitlab-ci-local --list-csv --variables-file ci/test/D-tag-preprod.variables.yml

How it works

1. Load the test case
2. Run gitlab-ci-local
3. Capture the CSV output listing all jobs that would run:

   name;stage;when;allowFailure;needs
   📦🌐api-build;📦 Package;on_success;false;[]
   🗄️✅back-unit-tests;✅ Test;on_success;false;[]
   ...

1. Compare with the reference file ci/test/D-tag-preprod.csv:
   • ✅ Match → Test passes
   • ❌ Diff → Show diff, fail test

When job name / stage / when condition / allowFailure flag / needed jobs change, the CI will let you know, and you will commit the change or fix the CI regression.

3. Setting up the test infrastructure

Directory structure

Create a ci/test/ directory to store test definitions and reference files:

ci/
└── test/
    [...]
    ├── D-tag-preprod.variables.yml      # Tag triggering preprod deployment
    ├── D-tag-preprod.csv                 # Expected jobs (reference)
    ├── F-scheduled.variables.yml         # Scheduled nightly pipeline
    └── F-scheduled.csv                   # Expected jobs (reference)

The naming convention {Type}-{Description}.variables.yml makes it easy to identify test scenarios.

Test case definition

Each .variables.yml file contains the GitLab CI predefined variables that simulate a specific pipeline context:

D-tag-preprod.variables.yml - Tag triggering preprod deployment:

CI_COMMIT_TAG: "v1.2.3"

F-scheduled.variables.yml - Scheduled nightly pipeline:

CI_PIPELINE_SOURCE: "schedule"
CI_COMMIT_BRANCH: "main"

The key is to set the variables that your workflow:rules and job rules: check to determine pipeline behavior.

Reference CSV files

The reference CSV files contain the expected job list. They're auto-generated on first run and then committed. Here's an example:

F-scheduled.csv:

name;description;stage;when;allowFailure;needs
🔎🖥️front-lint;"";🔎 Check;on_success;false;[]
✅🖥️front-unit-tests;"";🔎 Check;on_success;false;[]
📦🖥️front-build;"";📦 Package;on_success;false;[]
📦🌐api-build;"";📦 Package;on_success;false;[]
🗄️✅back-unit-tests;"";✅ Test;on_success;false;[]
🗄️🧩✅back-integration-tests;"";✅ Test;on_success;false;[]
🕵️💯security-scan;"";🕵 Quality;on_success;false;[]
🕵ᯤ🌐api-sonar;"";🕵 Quality;on_success;false;[📦🌐api-build]

This file serves as documentation and test oracle simultaneously:

• Developers can quickly see which jobs run for scheduled pipelines
• CI validates that reality matches expectations

4. The validation script

Here's a Bash script that automates the validation. It discovers test cases, runs gitlab-ci-local, and compares with reference files:

#!/bin/bash
# ci/test-ci-jobs-list.sh
#
# Tests CI non-regression by comparing gitlab-ci-local --list-csv
# output with committed reference files.
# Creates/updates reference files when differences are detected.
#
# Usage: ./ci/test-ci-jobs-list.sh [test-case-name]

set -e
FAILED=0
TOTAL=0

# Optional: run a single test case
if [ -n "$1" ]; then
    FILES="ci/test/$1.variables.yml"
else
    FILES="ci/test/*.variables.yml"
fi

for varsFile in $FILES; do
    [ -f "$varsFile" ] || continue

    baseName=$(basename "$varsFile" .variables.yml)
    referenceFile="ci/test/$baseName.csv"

    echo "🔍 Testing $baseName..."
    ((TOTAL++))

    # Generate current job list (filter out logs, keep only CSV)
    gitlab-ci-local --list-csv --variables-file "$varsFile" 2>/dev/null | \
        grep -E '^[^[]' | grep -v '^$' > "$referenceFile.tmp" || true

    # Ensure header is present
    if ! head -1 "$referenceFile.tmp" | grep -q "^name;"; then
        gitlab-ci-local --list-csv --variables-file "$varsFile" 2>&1 | \
            grep -E "^name;" > "$referenceFile.header"
        cat "$referenceFile.header" "$referenceFile.tmp" > "$referenceFile.new"
        mv "$referenceFile.new" "$referenceFile.tmp"
        rm -f "$referenceFile.header"
    fi

    mv "$referenceFile.tmp" "$referenceFile"

    # Check for git differences
    if ! git diff --exit-code -- "$referenceFile"; then
        echo "❌ FAILED: $baseName - jobs list has changed"
        ((FAILED++)) || true
    else
        echo "✅ PASSED: $baseName"
    fi
done

echo ""
echo "=========================================="
echo "Results: $((TOTAL - FAILED))/$TOTAL passed"
echo "=========================================="

if [ $FAILED -gt 0 ]; then
    echo "Review the diffs above and commit updated reference files."
    exit 1
fi

💡 Note: A PowerShell version is available on request for Windows-based runners.

5. The CI job

Add a job resembling this to your .gitlab-ci.yml:

✅🦊validate-ci-jobs-list:
  stage: ✅ Test
  resource_group: avoid-gcl-concurrency  # gitlab-ci-local may not be thread-safe
  tags: [your-runner-tag]
  image: node:20-alpine
  before_script:
    - npm install -g gitlab-ci-local
  script:
    - ./ci/test-ci-jobs-list.sh

Key points:

• resource_group: Prevent concurrent executions that might conflict
• rules:: you may want to run this on merge request when CI has changed, and on long-lived branches

6. Workflow: adding or modifying jobs

When you modify CI rules, follow this workflow:

1. Make your changes to .gitlab-ci.yml or included files
2. Run locally (optional but recommended):

   ./ci/test-ci-jobs-list.sh

1. Review the diffs - the script shows exactly what changed:

  🗄️✅back-unit-tests;"";✅ Test;on_success;false;[]
  🗄️🧩✅back-integration-tests;"";✅ Test;on_success;false;[]
− 🕵ᯤsonar;"";✅ Test;manual;true;[]
+ 🕵ᯤsonar;"";✅ Test;on_success;false;[]
  🕵ᯤ🌐api-sonar;"";🕵 Quality;on_success;false;[📦🌐api-build]

1. If intentional, commit the updated CSV files along with your CI changes
2. If unintentional, fix your rules before committing

This creates a self-documenting system: the git history of CSV files shows exactly when and why job behavior changed.

7. Testing with rules:changes

The examples above cover pipeline types where all jobs of a category run (protected branches, tags, scheduled). But on a real project with module-based rules: using changes:, MR pipelines only trigger jobs for modules whose files were modified. This is where things get spicy — and where regressions love to hide.

The problem with rules:changes and gitlab-ci-local

gitlab-ci-local does evaluate rules:changes — it has access to the git diff. But since our test job runs inside the MR pipeline, the diff contains whatever files happen to be modified in the current MR. A test for "backend-only pipeline" would suddenly include frontend jobs if someone touched a frontend file in the same branch. The results depend on the MR content, not on CI configuration — the opposite of a reliable test.

The solution is to guard changes: rules with $GITLAB_CI == "true":

.api-mr-rules:
  rules:
    - if: $CI_MERGE_REQUEST_ID && $GITLAB_CI == "true"
      changes:
        - src/Api/**/*
        - src/Commons/**/*

In real GitLab CI, $GITLAB_CI is always "true", so the rule works normally — changes: is evaluated against the MR diff. But gitlab-ci-local sets $GITLAB_CI to "false", which makes the if: condition fail before changes: is even evaluated. The entire rule is cleanly skipped, no ambiguity. Schrödinger's job: it exists in your YAML but never appears in the output — by design.

This is what makes test results deterministic: regardless of which files are modified in the current MR, the changes: rules are consistently neutralized, and only the label-based fallback (below) controls which jobs appear.

The force-build label workaround

The trick is to add a label-based bypass to every module's rules. In your actual CI, this serves double duty: developers use it when they need to force-rebuild a module (dependencies changed outside the changes: scope, cache issues, cosmic rays), and tests use it to simulate "this module has changes":

.api-mr-rules:
  rules:
    - if: $CI_MERGE_REQUEST_ID && $GITLAB_CI == "true"
      changes:
        - src/Api/**/*
        - src/Commons/**/*
    - if: $CI_MERGE_REQUEST_LABELS =~ /force-build-back/

In real CI, the first rule handles normal operation (job runs when matching files change). In gitlab-ci-local, the first rule is dead — only the label matters. Now the test variables file simply sets the right label:

A-MR-back.variables.yml — simulating backend changes:

CI_MERGE_REQUEST_ID: "12345"
CI_MERGE_REQUEST_LABELS: "force-build-back"

A-MR-front.variables.yml — simulating frontend changes:

CI_MERGE_REQUEST_ID: "12345"
CI_MERGE_REQUEST_LABELS: "force-build-front"

A-MR-no-change.variables.yml — the MR where only non-module files changed (docs, README, etc.):

CI_MERGE_REQUEST_ID: "12345"
CI_MERGE_REQUEST_LABELS: ""

This last one is particularly valuable: it verifies that common jobs (config generation, post-deploy checks) still run even when no module was touched. The CI equivalent of checking that the lights still work when nobody's home.

💡 We won't dive deeper into label-based pipeline control here — that topic deserves its own article. Stay tuned.

Per-module test cases

With this approach, the ci/test/ directory naturally reflects your module structure:

ci/test/
├── A-MR-no-change.variables.yml          # MR with no module changes
├── A-MR-no-change.csv
├── A-MR-back.variables.yml               # Backend module changes
├── A-MR-back.csv
├── A-MR-front.variables.yml              # Frontend module changes
├── A-MR-front.csv
├── A-MR-migrations.variables.yml         # Database migrations
├── A-MR-migrations.csv
├── A-MR-all-force-build-labels.variables.yml  # Everything activated
├── A-MR-all-force-build-labels.csv
├── B-protected-branch.csv                # Non-MR types (no changes: involved)
└── ...

The resulting CSV diffs tell a precise story. For instance, A-MR-back.csv might list 33 jobs while A-MR-front.csv lists only 18 — you can instantly verify that backend Sonar jobs don't sneak into frontend-only pipelines, and that shared deployment jobs appear in both.

On the project that inspired this approach (7 modules, 50+ jobs, 5 pipeline types), we ended up with 11 test cases covering every meaningful combination. The entire suite runs in under 10 seconds. That's less time than it takes to explain to a colleague why their MR pipeline is mysteriously empty.

8. Documentation that writes itself

The CSV reference files serve a dual purpose: automated validation and always up-to-date documentation.

If you use a doc-as-code tool like VitePress, Asciidoctor, or Docusaurus, you can render CSV files as HTML tables at build time — either natively or through a custom plugin. Most tools support CSV includes out of the box or with minimal effort, and any AI assistant can generate the glue code for your specific stack in seconds.

In your documentation markdown, it would look something like this:

## Pipeline Type D: Tag (preprod deployment)

```csv-table ci/test/D-tag-preprod.csv
```

## Pipeline Type F: Scheduled (nightly)

```csv-table ci/test/F-scheduled.csv
```

💡 Note: Adapt the csv-table syntax to your tool.

The same ci/test/*.csv files then serve two purposes:

1. CI Validation: gitlab-ci-local compares actual job lists against these reference files → pipeline passes or fails
2. Documentation build: your doc tool renders these CSV files as HTML tables → always accurate documentation

No more outdated documentation. No more "the wiki says X but CI does Y". The CSV files are the single source of truth — enforced by CI, displayed by docs, and immune to the corporate amnesia that plagues most wikis.

Wrapping up

This approach provides several benefits:

Benefit	Description
🛡️ Regression prevention	Catch unintended rule changes before they reach production
📖 Living documentation	CSV files document expected behavior for each pipeline type
🔍 Clear diffs	See exactly which jobs were added, removed, or modified
⚡ Fast feedback	Tests run in seconds, not minutes
🎯 Granular testing	Test specific pipeline variants independently

The key insight is treating your CI configuration as code that deserves its own tests. Just as you wouldn't ship application code without tests, you shouldn't ship CI changes without validating their effects.

This technique has saved countless hours of debugging "why doesn't this job run anymore?" issues. The upfront investment in setting up the test infrastructure pays off quickly as your CI configuration grows in complexity. Combined with other CI best practices to avoid widespread anti-patterns, it builds a solid foundation for maintainable pipelines.

Further reading

All Articles by Theme

Benoit COUETIL 💫

Benoit COUETIL 💫

Benoit COUETIL 💫

Follow

Feb 19

All Articles by Theme

#automation #devops #gitlab #kubernetes

2 reactions

Add Comment

7 min read

This article was enhanced with the assistance of an AI language model to ensure clarity and accuracy in the content, as English is not my native language.