DEV Community

Pavan Kumar Sadashiv
Pavan Kumar Sadashiv

Posted on

Eliminating Flaky Tests in GitLab CI/CD: Predictive Selection, Quarantining & JUnit Reports

#gitlab #cicd #devops #testing

Eliminating Flaky Tests in GitLab CI/CD: Predictive Selection, Quarantining & JUnit Reports

Few things erode engineering morale more than a flaky test.

You spend hours crafting an elegant, bug-free feature. You open your Merge Request, and 20 minutes later the pipeline turns red.

You inspect the failure, only to find that an unrelated asynchronous UI test timed out by 5 milliseconds. You click "Retry", and the pipeline passes.

In large engineering organizations, flaky tests cost thousands of wasted CI runner minutes and train developers to ignore legitimate build failures.

In GitLab Core (a codebase with over 150,000 unit and integration tests), maintainers developed sophisticated strategies to tame flakiness without slowing down velocity.

In this practical guide, I will show you how to implement automated retries with backoff, test quarantining, predictive test selection, and native JUnit XML test reporting in your GitLab CI/CD pipelines.

1. The Flaky Test Lifecycle in CI/CD 

+-------------------------------------------------------------+
|                      Test Execution Fails                   |
+-------------------------------------------------------------+
                              |
                              v
+-------------------------------------------------------------+
|                  Is failure transient? (retry)              |
|          GitLab automatically retries up to 2 times         |
|              (when: runner_system_failure, etc)             |
+-------------------------------------------------------------+
               |                               |
        (Still Failing)                    (Passed)
               v                               v
+-----------------------------+   +---------------------------+
|   Tag as :quarantine        |   |   Flag as Potential Flake |
| - Runs in non-blocking job  |   |   Log failure to metrics  |
| - Issues created to debug   |   +---------------------------+
+-----------------------------+
Enter fullscreen mode Exit fullscreen mode

2. Smart Retries: Don't Retry Every Failure Blindly

A common antipattern is setting retry: 2 unconditionally on an entire job. If your code has a syntax error, retrying it twice simply wastes 3x more runner compute.

GitLab allows conditional retries based on specific failure causes:

test_suite:
  stage: test
  image: ruby:3.2-alpine
  retry:
    max: 2
    when:
      - runner_system_failure   # Host crashed or Docker daemon choked
      - stuck_or_timeout_failure# Ephemeral network lag
      - script_failure          # Set only if running an internal flaky test tracker
  script:
    - bundle exec rspec
Enter fullscreen mode Exit fullscreen mode

This ensures true infrastructure glitches are automatically self-healed, while genuine code syntax errors fail fast.

3. Test Quarantining: Protect Your Main Branch

When a test is identified as intermittently flaky, it should not be allowed to block entire engineering teams while awaiting a refactor.

Instead of commenting out the test (which causes silent test rot), quarantine the test by moving it to an advisory, non-blocking CI job:

Step A: Tag the Flaky Test

In your test suite (RSpec, Jest, PyTest, or Go):

# spec/features/checkout_spec.rb
RSpec.describe 'User Checkout', :quarantine do
  it 'completes Stripe payment callback', quarantine: 'https://gitlab.com/org/proj/-/issues/1234' do
    # Known race condition currently being resolved
  end
end
Enter fullscreen mode Exit fullscreen mode

Step B: Run Quarantined Tests in an allow_failure: true Job 

# 1. Normal Test Job (Excludes quarantined specs)
unit_tests:
  stage: test
  script:
    - bundle exec rspec --tag ~quarantine

# 2. Quarantined Job (Runs flaky specs without blocking MR merges)
quarantined_tests:
  stage: test
  script:
    - bundle exec rspec --tag quarantine
  allow_failure: true # Keeps MR pipeline green even if this fails!
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
Enter fullscreen mode Exit fullscreen mode

Quarantined tests still execute, generating telemetry and logs for the debugging engineer, but zero developer merges are blocked.

4. Visualizing Failures with Native JUnit XML Reports

Parsing raw terminal output to find which of 800 tests failed is exhausting.

GitLab CI natively parses standard JUnit XML test reports and displays a dedicated Tests tab directly in the Merge Request widget:

run_tests:
  stage: test
  script:
    - pytest --junitxml=report.xml
  artifacts:
    when: always # Always upload reports, even if tests fail!
    reports:
      junit: report.xml
Enter fullscreen mode Exit fullscreen mode

What You See in GitLab:

  • An expandable "Test summary: 452 tests passed, 1 test failed" box.
  • The exact failure message, file name, and stack trace without downloading terminal logs.
  • Immediate historical indicators showing if this test has failed in previous runs.

5. Predictive Test Selection (Save 70% CI Time)

In GitLab Core (which I observed while contributing MR !260212), running all 150,000 unit tests on every single commit is impossible.

Instead, GitLab uses Predictive RSpec:

  1. It analyzes the git diff of the current commit (git diff origin/main...HEAD).
  2. If you only changed files in lib/mcp/handlers/, it dynamically filters the test suite to execute only spec/lib/mcp/ and associated integration specs.
  3. Full regression suites run only on nightly scheduled builds or when changes touch foundational database schemas.

You can implement basic predictive selection in any language using simple path matching:

CHANGED_FILES=$(git diff --name-only origin/main...HEAD)

if echo "$CHANGED_FILES" | grep -q "^src/billing/"; then
  echo "Running Billing Test Suite..."
  pytest tests/billing/
fi
Enter fullscreen mode Exit fullscreen mode

6. Key Takeaways

  1. Never Retry Blindly: Scope retries to runner_system_failure and stuck_or_timeout_failure.
  2. Quarantine, Don't Delete: Move unreliable tests to an allow_failure: true job with a linked GitLab issue URL.
  3. Always Output JUnit XML: Leverage artifacts:reports:junit for instant failure visibility in Merge Requests.

Resources & Links

How does your organization handle flaky integration tests? Let’s share strategies in the comments!

Top comments (0)