DEV Community

Sergey Kupletsky
Sergey Kupletsky

Posted on

GitLab CI: Needs vs Dependencies — A Practical Guide

#gitlab #ci #devops #cicd

When Gitlab first introduced concept of needs I started experimenting with it.
At first glance it seems like it should replace dependencies entirely, and some guides advise switching to needs as it should work "better". However, these two properties are different and cannot fully replace one another.

In documentation there is an advice "You should not combine dependencies with needs in the same job". But if you are using templates with different levels of nesting and want to modify or redefine something in them, then it may be difficult to avoid combining these two properties.

Unfortunately, the results of using them together are not always obvious. Therefore, I decided to explore all possible combinations and create some sort of a cheat sheet on how the needs and dependencies combinations work.

In this article, I will share my observations and provide examples to show how to use them correctly and what to avoid.

To begin with, let's review the more or less obvious things that are described in the documentation.

Dependence on Job Status

In GitLab CI, the dependency on the jobs completion status is determined using the when property or its equivalent rules:when.

By default, the job is executed only if all jobs of the previous stages are completed successfully. However, this behavior can be configured so that the job starts regardless of the status of previous jobs with when: always (in pipeline examples, both options will occur).

How dependencies Works

Dependencies define which artifacts should be available for a particular job, rather than "dependencies" in the usual sense (such as waiting for other jobs to be completed).

If dependencies are not specified in the job, all jobs from earlier stages are considered as sources of artifacts, and the job downloads all artifacts from them.

If you specify an empty array (dependencies: []), the job does not download any artifacts, but continues to wait for all jobs from the previous stages to be completed.

If you specify specific jobs in dependencies, the completion of all previous stages is still expected, but artifacts are loaded only from the specified jobs.

From the point of view of pipeline execution, dependencies are not so much "dependencies" as instructions for downloading artifacts, so if the artifacts of the job specified in dependencies have been deleted or expired in time, the current job will fail with an error.

Example of a Pipeline Execution

Pipeline Source Code

GitLab Documentation: Dependencies

How needs Works

Unlike dependencies, which focus on artifacts, needs defines the actual dependencies between the execution of jobs. This means that jobs with the specified needs will wait for these jobs to be completed, regardless of their stage (including the current one).

If you do not specify needs, the behavior will be standard and depend on other parameters.

The empty array needs: [] indicates that the job can be started as soon as the pipeline is created, without waiting for jobs to be completed in other stages.

If a specific job is specified in needs, the current job will wait for its completion, and download its artifacts by default. You can disable automatic downloading of artifacts by adding the artifacts: false option.

Example of a Pipeline Execution

Pipeline Source Code

GitLab Documentation: Needs

Combining needs and dependencies

Now we come to the key point — how pipeline works with different combinations of needs and dependencies. Let’s explore the behavior using specific examples.

Example of a Pipeline Execution

Pipeline Source Code

Empty dependencies and needs 

dependencies: []
needs: []
Enter fullscreen mode Exit fullscreen mode

The job starts immediately after the pipeline is created. No jobs are expected to be completed, no artifacts are downloaded. This is equivalent to needs: []

Empty dependencies and Specified needs 

dependencies: []
needs:
  - job: someJob
Enter fullscreen mode Exit fullscreen mode

The job waits for someJob to complete but does not download its artifacts, since dependencies is an empty array. Artifacts here are controlled only by the dependencies property, which has a higher priority in this case.

Matching dependencies and needs 

dependencies:
  - someJob
needs:
  - job: someJob
Enter fullscreen mode Exit fullscreen mode

or

dependencies:
  - someJob
needs:
  - job: someJob
    artifacts: true
Enter fullscreen mode Exit fullscreen mode

The job waits for someJob to complete and downloads its artifacts.

Jobs in dependencies and needs with Artifacts Disabled 

dependencies:
  - someJob
needs:
  - job: someJob
    artifacts: false
Enter fullscreen mode Exit fullscreen mode

The job waits for someJob to complete but does not download its artifacts.
Option artifacts: false in needs has a higher priority and blocks artifact downloads, even if the job is specified in dependencies.

Inconsistent dependencies and needs 

dependencies:
  - someJob1
needs:
  - job: someJob2
Enter fullscreen mode Exit fullscreen mode

Pipeline execution error. All jobs specified in dependencies must be present in needs. If not, the pipeline will not even start.

Example of a Pipeline Execution Error

More Jobs in needs than in dependencies 

dependencies:
  - someJob1
needs:
  - job: someJob1
  - job: someJob2
Enter fullscreen mode Exit fullscreen mode

The job waits for both someJob1 and someJob2 to complete but only downloads artifacts from someJob1, as specified in dependencies.
In this case dependencies limit which artifacts should be downloaded.

More Jobs in dependencies than in needs 

dependencies:
  - someJob1
  - someJob2
needs:
  - job: someJob1
Enter fullscreen mode Exit fullscreen mode

Pipeline execution error. All jobs specified in dependencies must be present in needs. If not, the pipeline will not even start.

Conclusions

  1. Waiting for jobs to be completed is based on the needs property.
  2. To download artifacts, a job must be specified in both dependencies and needs.
  3. Option artifacts: false in needs has a higher priority and blocks artifact downloads, even if the job is specified in dependencies.
  4. If the job is specified in dependencies but is missing in needs, the pipeline will not start.

If you liked this article, you can support me with PayPal or follow me in:

Top comments (4)

Collapse
 
niranjan_velakanti_f6908f profile image
Niranjan Velakanti •

Thank you for the detailed and crystal-clear documentation.

Special thanks for your efforts in presenting it in such a well-structured and easy-to-understand manner.

Collapse
 
zavoloklom profile image
Sergey Kupletsky •

Thank you, I appreciate it!

Collapse
 
krzysztofcislo profile image
Krzysztof Cisło •

Thanks for the great article!

There is one more configuration which could be explained. Just needs without dependencies:

needs:
  - job: someJob
    artifacts: true
Enter fullscreen mode Exit fullscreen mode


`

Will job wait for someJob? Will artifacts be downloaded in this case if there is no dependencies section.

Collapse
 
karasu profile image
Apollyon •

Will job wait for someJob? Will artifacts be downloaded...
The answer is yes and yes, and that's why some articles claim that needs can totally replace all dependencies.