When Gitlab first introduced concept of needs I started experimenting with it.
At first glance it seems like it should replace dependencies entirely, and some guides advise switching to needs as it should work "better". However, these two properties are different and cannot fully replace one another.
In documentation there is an advice "You should not combine dependencies with needs in the same job". But if you are using templates with different levels of nesting and want to modify or redefine something in them, then it may be difficult to avoid combining these two properties.
Unfortunately, the results of using them together are not always obvious. Therefore, I decided to explore all possible combinations and create some sort of a cheat sheet on how the needs and dependencies combinations work.
In this article, I will share my observations and provide examples to show how to use them correctly and what to avoid.
To begin with, let's review the more or less obvious things that are described in the documentation.
Dependence on Job Status
In GitLab CI, the dependency on the jobs completion status is determined using the when property or its equivalent rules:when.
By default, the job is executed only if all jobs of the previous stages are completed successfully. However, this behavior can be configured so that the job starts regardless of the status of previous jobs with when: always (in pipeline examples, both options will occur).
How dependencies Works
Dependencies define which artifacts should be available for a particular job, rather than "dependencies" in the usual sense (such as waiting for other jobs to be completed).
If dependencies are not specified in the job, all jobs from earlier stages are considered as sources of artifacts, and the job downloads all artifacts from them.
If you specify an empty array (dependencies: []), the job does not download any artifacts, but continues to wait for all jobs from the previous stages to be completed.
If you specify specific jobs in dependencies, the completion of all previous stages is still expected, but artifacts are loaded only from the specified jobs.
From the point of view of pipeline execution, dependencies are not so much "dependencies" as instructions for downloading artifacts, so if the artifacts of the job specified in dependencies have been deleted or expired in time, the current job will fail with an error.
Example of a Pipeline Execution
GitLab Documentation: Dependencies
How needs Works
Unlike dependencies, which focus on artifacts, needs defines the actual dependencies between the execution of jobs. This means that jobs with the specified needs will wait for these jobs to be completed, regardless of their stage (including the current one).
If you do not specify needs, the behavior will be standard and depend on other parameters.
The empty array needs: [] indicates that the job can be started as soon as the pipeline is created, without waiting for jobs to be completed in other stages.
If a specific job is specified in needs, the current job will wait for its completion, and download its artifacts by default. You can disable automatic downloading of artifacts by adding the artifacts: false option.
Example of a Pipeline Execution
Combining needs and dependencies
Now we come to the key point — how pipeline works with different combinations of needs and dependencies. Let’s explore the behavior using specific examples.
Example of a Pipeline Execution
Empty dependencies and needs
dependencies: []
needs: []
The job starts immediately after the pipeline is created. No jobs are expected to be completed, no artifacts are downloaded. This is equivalent to needs: []
Empty dependencies and Specified needs
dependencies: []
needs:
- job: someJob
The job waits for someJob to complete but does not download its artifacts, since dependencies is an empty array. Artifacts here are controlled only by the dependencies property, which has a higher priority in this case.
Matching dependencies and needs
dependencies:
- someJob
needs:
- job: someJob
or
dependencies:
- someJob
needs:
- job: someJob
artifacts: true
The job waits for someJob to complete and downloads its artifacts.
Jobs in dependencies and needs with Artifacts Disabled
dependencies:
- someJob
needs:
- job: someJob
artifacts: false
The job waits for someJob to complete but does not download its artifacts.
Option artifacts: false in needs has a higher priority and blocks artifact downloads, even if the job is specified in dependencies.
Inconsistent dependencies and needs
dependencies:
- someJob1
needs:
- job: someJob2
Pipeline execution error. All jobs specified in dependencies must be present in needs. If not, the pipeline will not even start.
Example of a Pipeline Execution Error
More Jobs in needs than in dependencies
dependencies:
- someJob1
needs:
- job: someJob1
- job: someJob2
The job waits for both someJob1 and someJob2 to complete but only downloads artifacts from someJob1, as specified in dependencies.
In this case dependencies limit which artifacts should be downloaded.
More Jobs in dependencies than in needs
dependencies:
- someJob1
- someJob2
needs:
- job: someJob1
Pipeline execution error. All jobs specified in dependencies must be present in needs. If not, the pipeline will not even start.
Conclusions
- Waiting for jobs to be completed is based on the
needsproperty. - To download artifacts, a job must be specified in both
dependenciesandneeds. - Option
artifacts: falseinneedshas a higher priority and blocks artifact downloads, even if the job is specified independencies. - If the job is specified in
dependenciesbut is missing inneeds, the pipeline will not start.
If you liked this article, you can support me with PayPal or follow me in:
Top comments (2)
Thank you for the detailed and crystal-clear documentation.
Special thanks for your efforts in presenting it in such a well-structured and easy-to-understand manner.
Thank you, I appreciate it!