Wen Ping

Posted on Oct 31 Edited on Oct 31 Originally published at anguskit.com

The runner is not taking jobs: labels, online status, or concurrency do not match

#git #angusgit #runner

This article explains how to see which layer is stuck when a pipeline has already started and the job stays Pending. After reading it, you can use the wait reason next to the job to check whether the runner is online, its execution mode, its labels, its resources, and its concurrency, and decide whether to change the job or the runner. A runner is the program installed on a build machine that picks up a job and runs it.

Pipelines and runners are available on Team and Enterprise. Community has no CI/CD menu, and the steps in this article do not apply there.

If the list has no runner at all, sign in to AngusGit, open CI/CD → Runners in the sidebar, and click Add Runner. Change Control plane URL to an address the build machine can reach, name it for example build-linux-01, and click Generate token (it lasts 1 hour and is shown only once). When the job should run in a container, open the Run with Docker tab and set the labels to linux,docker. When the job should run directly on the build machine, open the Zip install tab, leave docker out of the labels, and install JDK 21 on the machine first. Copy the command onto the build machine and run it. After the list shows Idle, go back to the pipeline and read the wait reason.

1. How a runner is chosen

Each job uses runner in the YAML to say what kind of runner it needs, for example:

pipeline:
  build:
    runner:
      mode: DOCKER
      image: repo.anguskit.com/docker-public/angus-ci-maven:21
      labels: ["linux", "docker"]
    steps:
      - name: Build
        run: mvn -B -ntp clean package -DskipTests

AngusGit filters every runner in the order below. When a layer leaves zero runners, the job stops on that layer, and the wait reason names that layer:

Online: only runners whose status is Online, Idle, or Busy. Offline and Disabled are left out.
Execution mode: when the job writes mode, the runner’s mode has to be the same. The runner’s mode is fixed the first time it is connected, from its labels: a docker label means DOCKER, and no docker label means NATIVE. DOCKER runs the job in a container the runner starts from the image. NATIVE runs the job directly on the build machine. When the job omits mode, this layer does not filter.
Labels: the runner must have every label in the job’s labels. Extra labels on the runner are fine.
Resources: when the job writes resources, the CPU and memory the runner reports have to be enough.
Concurrency: the number of jobs already on the runner has to be lower than its Max concurrency.

Among the runners that remain, AngusGit picks the one that is currently the most idle.

2. Read the wait reason first

Open CI/CD → Pipelines, click the pipeline name, and in Runs click the number of this run to open the run detail. On the Jobs tab, a job that is Pending shows a reason beside it. The next five sections match the five reasons.

When the labels do not match, the reason is “No Runner with labels linux,docker”. When the mode does not match, it is “No Runner matching mode DOCKER”. When none are online, it is “No online Runner”. When resources are short, it is “No Runner matching resources ({resources})”, with the job’s requirement inside the parentheses. When concurrency is full, it is “Matching Runners are at capacity; waiting for a free slot”. On the current instance, dev-1 and priv-1 are both idle, so the page has no job sitting on Pending. The reason text is still those sentences. When yours matches, use the section for that sentence.

You can catch one case before you trigger. After you click Trigger, if there is no runner that is online, idle, or busy, the page shows No schedulable runner. The explanation is “There is no online or busy runner. A trigger would stay queued. Connect at least one runner first.” Next to it is Set up a runner.

3. No online Runner

When the wait reason is “No online Runner”, every runner is Offline or Disabled.

Offline: the runner sends a heartbeat every 15 seconds. If more than 90 seconds pass with no heartbeat and the connection is also gone, it becomes Offline. Usual causes are a powered-off build machine, a runner process or container that exited, or a broken network path from the build machine to AngusGit. On the build machine: for the Docker method, docker ps shows whether the container is still there; if it is gone, docker start <container name>. For the Zip method, go to /opt/AngusRunnerAgent and run ./startup-agent.sh again.
Disabled: someone clicked Disable in the list. The confirmation says “Disable runner "{name}"? It will stop picking up jobs, and heartbeats will not bring it back online.” Bringing the machine back does not help. Click Enable in the list.
There is no runner at all: none has been connected yet. Follow the start of this article: under CI/CD → Runners, click Add Runner, copy the command onto the build machine, and wait until the list shows Idle.

4. No Runner matching mode …

When the reason says DOCKER, there is no online runner with the docker label. When it says NATIVE, every online runner has the docker label. A DOCKER job runs in a container the runner starts from the image. A NATIVE job runs directly on the build machine (the host).

What the job writes	Runner it needs	Usual situation
`mode: DOCKER`	Has the `docker` label	Only a host runner was connected with Zip, and its labels do not include `docker`
`mode: NATIVE`	Does not have the `docker` label	Only the Docker method was used, and the default labels are `linux,docker`

There are two fixes: change the job’s mode so it matches the runners you already have, or connect another runner of the matching mode.

The runner’s mode is fixed only on the first connect. Adding or removing the docker label later in Edit runner changes the label. The mode stays as it was. The label hint there is “Comma-separated. Labels only select a runner and do not change its execution mode.” To change the mode, connect a new runner with the matching method.

When one machine needs both modes, install two runners on it: one connected with Docker, and one connected with Zip whose labels do not include docker.

mode is the uppercase word DOCKER or NATIVE. Which runner each mode needs is in the table above in this section. If that runner is not there yet, connect one from the start of this article, using the matching tab.

5. No Runner with labels …

Every label in the job’s labels has to appear on the same runner. The reason lists the labels the job requires. The label text is joined with commas and no space, as in linux,docker.

Compare them one by one. On CI/CD → Runners, read the labels of each online runner. Three mismatches are common:

One label is missing. The job wants ["linux", "docker", "jdk21"], and the runner only has linux,docker.
The case differs. Linux and linux are two different labels.
The spelling differs. The job writes x86, and the runner writes amd64.

Changing a runner’s labels does not require a reinstall. Open Edit runner from the list, change Labels, and save. Or change the labels in the job YAML so they match the runner. The docker label is the exception: it is tied to the execution mode. See the previous section.

6. No Runner matching resources

The job’s runner.resources asks for CPU or memory, and no online runner reports a total that meets it. The parentheses after the reason show what the job asked for, for example cpu>=4 memory>=8192MB.

Open the runner detail from CI/CD → Runners to see that machine’s CPU and memory. Lower the job’s resource request to a range the machine can meet, or connect a larger machine. If you do not need a resource limit, remove resources.

7. Matching Runners are at capacity; waiting for a free slot

This is not a wrong config. Every runner that can take this job is busy. The number of jobs on each of them has reached Max concurrency, and this job is in the queue. It starts on its own when an earlier job finishes.

When the queue is a regular event, there are two moves:

Raise concurrency. In Edit runner, raise Max concurrency. The range is 1 to 64. Look at CPU and memory on the runner detail first. If the machine is already busy, a higher number only makes every job slower. The hint is “Takes effect for scheduling immediately. Online runners receive the change without a restart.”
Connect another one. Connect another runner with the same labels. The job is sent to the one that is more idle.

8. The job was picked up, then went back to Pending

The job had started, and a while later it is Pending again. The usual cause is that the runner running it lost the connection in the middle. AngusGit sees the disconnect, puts the jobs that were running on that runner back on the queue, and picks a runner again. When only that one runner can take the job and it has not recovered, the wait reason becomes No online Runner. Check that machine as in section 3.

Common questions

The job does not write runner. Where is it sent?

When both mode and labels are omitted, any online runner that still has a free concurrency slot may take it. The job also stays out of a container and runs directly in the runner’s environment. To keep the result predictable, write mode and labels on every job.

Can one pipeline have both DOCKER and NATIVE jobs?

Yes. Each job writes its own runner, and AngusGit picks a runner per job. Both kinds of runner have to be online.

I just changed a runner’s labels. Will a job that is already waiting be picked up on its own?

Yes. A pending job is matched again after a short interval. Once the labels match, it is picked up. You do not trigger again. Wait a moment and refresh the run detail.

Summary

When a job stays Pending, read the wait reason on the run detail first. AngusGit checks online status, execution mode, labels, resources, and concurrency, in that order. The reason names the layer that filtered every runner out. Recover an offline runner on the build machine. For a disabled runner, click Enable. The execution mode is decided by the docker label when the runner is connected: a DOCKER job needs a runner that has it, and a NATIVE job needs a runner that does not. Every label on the job has to be on the same runner, with the same case. A resource request has to stay within the machine’s total. A full concurrency slot means the job is queued: wait, or add capacity.

Conclusion: When a job stays Pending, use the wait reason beside the run detail to check whether the runner is Online, which execution mode it got at connect time (DOCKER / NATIVE), whether it has every labels value the job asks for (case-sensitive), and whether resources and concurrency slots still have room.

DEV Community