The runner that was busy on someone else's job

My CI pipeline sat pending for ten minutes. The runner was healthy. So was every other pipeline in the queue. The default setting was the bug.

Healthy and stuck at the same time

GitLab Runner ships with concurrent = 1 in /etc/gitlab-runner/config.toml. That one line quietly makes every pipeline run single-file across every project sharing that runner. Not per project. Across all of them. So a chatty repo pushing every few minutes can starve everything else, and the symptom is the least helpful one possible: your job sits pending while gitlab-runner status cheerfully reports "healthy."

Healthy and stuck at the same time is the tell. The runner is not broken. It is busy, on something that is not yours.

The diagnosis

The check that confirmed it runs inside the runner's container:

docker ps --filter name=runner-
docker inspect <build-container> --format '{{.Config.Env}}'

Read CI_PROJECT_PATH and CI_PIPELINE_ID off the running build container and you learn exactly which project is holding the single lane. In my case it was a different repo entirely, mid-build, with mine queued politely behind it.

The fix and the trade

The fix is one number: bump to concurrent = 2, or higher, watching RAM, because every extra lane is another set of job containers running at once. Back up the config first, restart the runner, and the pending pipelines unblock within about thirty seconds.

The broader lesson is about defaults. concurrent = 1 is tuned for one project per runner, which is the simplest possible case and rarely the real one. The moment your runner serves several repos, that default stops protecting you and starts throttling you. "Healthy but not moving" is the signature of a default you have outgrown, not a component that has failed.