If you migrate your multirepo to a monorepo, or if your project is getting big enough to consider running only part of continuous integration (CI) - then it can make sense to run only those parts of CI that could have been affected by the change. This article will show how to achieve it on the GitLab platform, using a simple repository as an example.
The approach
The project is split into folders. For distinguishing if a given part of the project was modified, we use rules:changes so every part of the project that we want to be able to run in separation from the rest should be placed in one folder.
My assumptions are as follow:
- you want to run only changed sections of CI in the merge requests (MR) - mainly to save CI resources. So we can avoid running whole 30 minutes of jobs for a change we know is not likely to affect them. This is especially important if we have code in the repo that is not depending on each other - for example, our main application in one place and some landing pages in other folders.
- after changes are merged to master/main, we want to build everything no matter if it was changed or not. In this way, our main branch is indeed continuously integrated, and we keep on checking on even less commonly changed parts of the project.
Configuration
My project structure is simple:
$ git ls-files
.gitlab-ci.yml
README.md
backend/README.md
frontend/README.md
I have 2 folders, backend
& frontend
. Each would host files of a given part of our project. This approach scales for any number of sub-projects - we could have company-website
, slack-bot
, or whatnot inside.
.gitlab-ci.yml
step by step
The configuration starts with defining the stages:
stages:
- build
- test
- deploy
This one is copied from GitLab's starting CI template. We can customize it with adding or removing stages. As we define needs:
, there is no speed penalty for adding more stages - each job is executed as soon as its requirements are defined in needs:
are met. For example, in my project, I ended up adding pre-build
to run some preparation scripts before building docker images in my project.
variables:
RULES_CHANGES_PATH: "**/*"
The default value for our changes configuration - by default, the job that extends our base config will be executed for any changes.
.base-rules:
rules:
...
Our base config. I define it in a way that requires us to add it with extends: .base-rule
- probably we could define those rules on the top level, but it's something a headache to configure everything in a way that works as expected in every case. I found it easier to have control over if the .base-rules
are set or not.
.base-rules:
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: always
...
The rules
are checked in order. Our 1 rule - if it's master/main, CI should always run the job.
- if: '$CI_PIPELINE_SOURCE == "push"'
when: never
Here, we avoid duplicated jobs for merge requests. Without, GitLab would create 1 pipeline for the branch and a "detached pipeline" for the MR. As the branch pipeline doesn't seem to support changes:
, we disable branch one & delay starting CI until an MR is created.
- if: $CI_COMMIT_TAG
when: never
Similarly, we don't need CI running for a tag.
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- $RULES_CHANGES_PATH
In MRs, we start jobs for when there are changes in the path defined in the RULES_CHANGES_PATH
variable.
- when: manual
allow_failure: true
Otherwise, we all the job to be triggered manually.
.frontend
& .backend
Now, we define 2 more jobs to be extended from:
.backend:
extends: .base-rules
variables:
RULES_CHANGES_PATH: "backend/**/*"
.frontend:
extends: .base-rules
variables:
RULES_CHANGES_PATH: "frontend/**/*"
In this way, we avoid duplicating the same path definition in each job we define for one or the other part - a possible source of errors.
Example jobs
On top of that all, we can define our jobs as:
backend-build:
stage: build
extends: .backend
needs: []
script:
- echo "Compiling the backend code..."
frontend-build:
stage: build
extends: .frontend
needs: []
script:
- echo "Compiling the frontend code..."
Complete .gitlab-ci.yml
So, in the end, the complete config files are like this:
stages:
- build
- test
- deploy
variables:
RULES_CHANGES_PATH: "**/*"
.base-rules:
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: always
- if: '$CI_PIPELINE_SOURCE == "push"'
when: never
- if: $CI_COMMIT_TAG
when: never
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
changes:
- $RULES_CHANGES_PATH
- when: manual
allow_failure: true
.backend:
extends: .base-rules
variables:
RULES_CHANGES_PATH: "backend/**/*"
.frontend:
extends: .base-rules
variables:
RULES_CHANGES_PATH: "frontend/**/*"
backend-build:
stage: build
extends: .backend
needs: []
script:
- echo "Compiling the backend code..."
frontend-build:
stage: build
extends: .frontend
needs: []
script:
- echo "Compiling the frontend code..."
backend-test:
stage: test
extends: .backend
needs: ["backend-build"]
script:
- echo "Testing the backend code..."
frontend-test:
stage: test
extends: .frontend
needs: ["frontend-build"]
script:
- echo "Testing the frontend code..."
backend-deploy:
stage: deploy
extends: .backend
needs: ["backend-test"]
script:
- echo "Deploying the backend code..."
frontend-deploy:
stage: deploy
extends: .frontend
needs: ["frontend-test"]
script:
- echo "Deploying the frontend code..."
Working CI
With a setup like this, you can have your backend CI run for backend MR:
frontend, for MR with frontend changes:
and each commit will trigger all CI jobs once it's merged to the main branch:
Refrences
You can find the repo I used to write this article here.
Summary
In this article, we have seen how to set up partially split CI for branches in GitLab. Please leave a comment if you find this article helpful or have some questions about this approach.
Top comments (0)