Once we had a gitlab CI file. It was short and sweet. One year later, it grew 350 lines long.
There were these problems:
- Too much content - too much scrolling, hard to visualize and work on.
- Hard to disable some jobs temporarily - mostly for debugging infra or test environment or emergency deployment (let that never happen!).
Let's try to solve it with some features from gitlab.
Gitlab CI supports using templates within the
Consider a sample
.gitlab-ci.yml file as follows
stages: - setup - soft-qa //lint and unit tests - build - hard-qa //e2e's - deploy-storybook - pack - notify-devs-staging-can-be-deployed - deploy-staging - notify-devs-prod-can-be-deployed - deploy-production - suggest-release-notes variables: var1: '1' // .. and so on conditions: only_master: // configs branches: // configs // and more... ## cache related configs ## setup related configs ## jobs for each for the stages start . . . . // 200 lines later . ## jobs end
I will still keep the stages in the
.gitlab-ci.yml untouched so that I can get a preview of all the stages right at the start of the file.
Let's now split this into smaller templates.
For templates, I will create a folder in the repo root called
ci-templates. Now let's extract out one job from the main file and place it in a template in this folder. Note, all these files has to YAML, to be included into a
/ci-templates/.soft-qa.yml soft-qa: image: node:14.5 stage: soft-qa <<: *npm_cache_pull allow_failure: false script: - yarn lint - yarn test:unit:ci artifacts: paths: - coverage/lcov.info
Time to use the template. Go to
.gitlab-ci.yml and include this file like below. I will place it at the position of the replaced job.
We are using this syntax - with relative path - as we are using the file from the same repo. You can keep the file in another repo on the same gitlab instance or even in a public remote repository and use it! Just look up the syntax and update accordingly, like below:
For another repo:
include: - project: 'my-space/my-another-project' file: '/templates/.build-template.yml'
include: - remote: 'https://somewhere-else.com/example-project/-/raw/master/.build-template.yml'
In a similar way, extract out other jobs as well and remove them from the .gitlab ci file. I have abstracted the notify stages to a file called
.notifications.yml and deployment related jobs to
.deploy.yml, thus separating the concerns from one single file. Now, include becomes a list like below:
include: - '/ci-templates/.soft-qa.yml' - '/ci-templates/.build.yml' - '/ci-templates/.hard-qa.yml' - '/ci-templates/.deploy-storybook.yml'
When pipeline starts, all included files are evaluated and deep merged into the .gitlab-ci file.
Things are getting interesting. I have certain conditions like only_tag, only_branches in these jobs. How would I provide them to these files without duplicating it in every files?
I will consolidate my conditions in a file, say,
// /ci-templates/.conditions.yml .only_tag: only: - /^v\d+\.\d+\.\d+$/ // and more...
To use it in a template, include it first in the .gitlab-ci file
include: - '/ci-templates/.conditions.yml' - '/ci-templates/.soft-qa.yml' - '/ci-templates/.build.yml' - '/ci-templates/.hard-qa.yml' - '/ci-templates/.deploy-storybook.yml'
Then, in the template file, add extend. Extend is a way of extending the congfigurations to another file. YAML anchors can be used but only from the same file. So this is a way to use jobs from another template.
deploy_staging: extends: .only_tag <<: *deploy
Now, deploy_staging will be created only for tags.
Finally, the .gitlab-ci file will look something like:
stages: - setup - soft-qa //lint and unit tests - build - hard-qa //e2e's - deploy-storybook - pack - notify-devs-staging-can-be-deployed - deploy-staging - notify-devs-prod-can-be-deployed - deploy-production - suggest-release-notes variables: var1: '1' // .. and so on ## cache related configs ## setup related configs include: - '/ci-templates/.conditions.yml' - '/ci-templates/.soft-qa.yml' - '/ci-templates/.build.yml' - '/ci-templates/.hard-qa.yml' - '/ci-templates/.deploy-storybook.yml' - '/ci-templates/.pack.yml' - '/ci-templates/.notifications.yml' - '/ci-templates/.deploy.yml' - '/ci-templates/.suggest-release-notes.yml'
Now let's go back to the initial problems and check whether they are fixed:
- Too much content to scroll - now, we can view whole content in the IDE window - Fixed
- Hard to disable jobs - now, we just have to comment the template in the include list - Fixed
How will I know, if I commented out a required job? Like, by mistake, I commented the pack job. For deployment packed resources are necessary. Here, we will know that the deployment failed much later when the deployment runs. To solve this, there is a way to mark dependencies on previous jobs. That is explained in next step.
One way to track that you have the dependent artifacts are available before starting a job is by using
By default, all artifacts from previous stages are passed to each job. However, you can use the dependencies keyword to define a limited list of jobs to fetch artifacts from.
build_frontend: stage: build script: yarn build deploy: stage: deployment dependencies: - build_frontend
In this case, if the build_frontend is not available, while merging the templates, the pipeline will report error that the
build_frontend is not available. Easy to understand.
Another utility is the
needs. It is mainly used when you are running jobs out of order and still ensure that the dependency job is completed before starting a job. The difference of
dependencies is that, in
needs, no artifacts from previous steps are downloaded by default. You need to use
artifacts: true config on the job like below to download them:
deploy: needs: - job: build_1 artifacts: true - job: build_2 artifacts: false
This helped me reduce the complexity of the CI file. Hope it helps you too.