DEV Community

Manish Kumar
Manish Kumar

Posted on

Fixing Silent Misconfigurations in GitLab Runner: Adding Concurrency Alias Support

When operating self-hosted or cloud infrastructure for CI/CD pipelines, one subtle configuration mistake can cause unexpected resource exhaustion or build instability.

In my recent contribution to GitLab Runner (MR !7782), I solved a long-standing developer trap related to TOML configuration parsing that previously led to runners processing unlimited parallel jobs unintentionally.

Here is the story of how a small alias mapping prevents critical operational failures in production CI/CD environments.


The Problem: The "Concurrency" Trap

GitLab CI/CD documentation and community discussions frequently use the word "concurrency" to describe how many jobs a runner or worker pool can execute simultaneously. Furthermore, within a runner's specific configuration, settings like request_concurrency already use that exact terminology.

Consequently, DevOps engineers and system administrators frequently wrote:

concurrent = 1

[[runners]]
  name = "docker-runner"
  url = "https://gitlab.com"
  concurrency = 1  # ⚠️ Expected: Only run 1 job at a time
  executor = "docker"
Enter fullscreen mode Exit fullscreen mode

Or at the global level:

concurrency = 4    # ⚠️ Expected: Maximum 4 global concurrent jobs
Enter fullscreen mode Exit fullscreen mode

What actually happened?

Under the hood, GitLab Runner uses BurntSushi/toml to unmarshal config.toml into Go structs. Because Go struct tags strictly mapped:

  • concurrent to global Config.Concurrent
  • limit to runner-level RunnerConfig.Limit

Any key named concurrency was silently ignored without raising an error or warning!

Because limit defaulted to its zero value (0), and 0 in GitLab Runner represents unlimited, a runner meant to strictly execute one build at a time would suddenly pick up and run dozens of jobs in parallel. This often resulted in Out-Of-Memory (OOM) errors, CPU throttling, and failed deployments.


The Solution: Intelligent Alias Fallback

In Merge Request !7782, we implemented alias handling during configuration loading:

  1. Global Level: If concurrent is not set but concurrency is present in the configuration, concurrency is mapped directly to Config.Concurrent.
  2. Runner Level ([[runners]]): If limit is not explicitly set, concurrency is mapped directly to RunnerConfig.Limit.
  3. 100% Backward Compatibility: Canonical keys (concurrent and limit) take priority if both are specified, ensuring existing configurations remain unaffected.
# Now fully supported and safe!
concurrency = 4

[[runners]]
  name = "production-runner"
  url = "https://gitlab.com"
  concurrency = 1   # Enforces single-job execution as expected
  executor = "docker"
Enter fullscreen mode Exit fullscreen mode

Verifying with Table-Driven Tests in Go

To ensure this fallback works reliably across all combinations of configurations, we added comprehensive table-driven tests (TestLoadConfig_ConcurrencyAlias):

  • Scenario A: Only concurrency specified at root level → Maps to Concurrent.
  • Scenario B: Only concurrency specified in [[runners]] → Maps to Limit.
  • Scenario C: Both concurrent/limit and concurrency present → Canonical fields take precedence.
  • Scenario D: Unset values retain their standard defaults.
go test -v ./common -run '^TestLoadConfig_ConcurrencyAlias$'
Enter fullscreen mode Exit fullscreen mode

All test cases pass cleanly, safeguarding runner stability across minor and major upgrades.


Key Takeaways

  1. Developer UX matters in Infra Configs: If operators commonly guess a configuration key based on natural naming conventions or documentation phrasing, supporting it as an alias eliminates dangerous failure modes.
  2. Beware of Silent Unmarshaling: When unmarshaling TOML/YAML/JSON in Go, unused fields don't always produce syntax errors, which can leave critical limits at their uninitialized zero values.
  3. Open Source Contribution: Small, well-targeted improvements can resolve issues open for years (Issue #4793) and prevent real-world production outages.

Check out the merged implementation on GitLab:

👉 GitLab Runner MR !7782


Author: Manish Kumar

GitLab: @manishpatel00

Top comments (0)