- Initial thoughts
- 1. Use a local docker image in early stages of job creation
- 2. Use the pipeline editor for minor YAML changes
- 3. Use gitlab-ci-local to run pipelines locally
- 4. Declare test branches to simulate long-lived branches and tags
- Wrapping up
- Further reading
Initial thoughts
As developers and CICD engineers, we are all too familiar with the time-consuming process of iterating on GitLab CI YAML modifications. It can be a frustrating cycle that involves multiple steps and setbacks:
- Make a change to the YAML file.
- Commit and push the changes.
- Verify that the CI YAML is valid and the graph is as expected.
- If there are any issues or errors, start the process all over again with a sense of despair.
- Wait for the entire pipeline to finish running on the current branch.
- If the pipeline fails or doesn't behave as desired, start over again with a heavy heart.
- Refactor commits to maintain a clean git graph and push with force.
- Merge multiple times to observe the behavior on long-lived branches and tags.
- If there are any issues with the merged code, start the process all over again with a sigh.
What we initially thought would only take a few minutes ends up consuming multiple days as we continuously go back and forth, all while trying to juggle other development tasks.
In this article, we aim to alleviate this pain by providing several tips that will make your life easier. Our goal is to transform the experience of tweaking your favorite pipelines from a tedious chore to an enjoyable task. So let's dive in and discover how you can streamline your GitLab CI YAML modifications!
1. Use a local docker image in early stages of job creation
When starting a new job with a new Docker image, it's natural to feel uncertain. Especially when using initially unknown public images, following the best practice Start CI with versioned public CI docker images. To accelerate the feedback loop and gain more confidence, we can try our commands directly in a local container.
Let's consider a NodeJS job as an example. The DockerHub image alpine/k8s seems to be the most suitable base image.
Before diving into the GitLab CI YAML, let's start an interactive container:
docker run -it --privileged=true node:20.5-alpine3.17 /bin/sh
Status: Downloaded newer image for node:20.5-alpine3.17
/#
You may wonder what shell GitLab is using in containers. In fact, it is a discovery process starting with bash, so best guess is to try /bin/bash and fallback to /bin/sh.
Now that you're inside the image, you can easily prepare and test your commands.
Although the base is alpine, let's assume we do not have that knowledge and want to use envsubst. If it's not available, we can try to install it with apt-get update && apt-get install envsubst:
/# envsubst --help
/bin/sh: envsubst: not found
/# apt-get update && apt-get install envsubst
/bin/sh: apt-get: not found
Since envsubst and apt-get are not available, it must be the alpine Linux package manager, apk:
/# apk --update add --no-cache envsubst
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/aarch64/APKINDEX.tar.gz
ERROR: unable to select packages:
envsubst (no such package):
required by: world[envsubst]
Ah! It seems that envsubst cannot be installed this way. A quick internet search reveals that the package is actually named gettext:
/apps# apk --update add --no-cache gettext
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.17/community/aarch64/APKINDEX.tar.gz
(1/9) Installing libgomp (12.2.1_git20220924-r4)
(8/9) Installing libxml2 (2.10.4-r0)
(9/9) Installing gettext (0.21.1-r1)
Executing busybox-1.35.0-r29.trigger
OK: 18 MiB in 26 packages
/#
/# envsubst --help
Usage: envsubst [OPTION] [SHELL-FORMAT]
Substitutes the values of environment variables.
By going through this discovery process within a few minutes, we saved ourselves from waiting for the pipeline to fail, which could have taken several hours for a job, especially positioned in last stages.
2. Use the pipeline editor for minor YAML changes
The GitLab CI/CD Pipeline Editor is a powerful web-based tool that provides a visual representation of your pipeline configuration. Accessible from the GitLab UI, the Pipeline Editor allows you to interactively modify your .gitlab-ci.yml file directly in the browser.
The Pipeline Editor offers several advantages. Before even commiting changes, it provides real-time feedback on the validity of your YAML syntax and helps prevent common errors. Additionally, it assists in exploring available keywords and includes inline documentation for GitLab CI/CD features. This visual approach can significantly streamline the process of making and validating changes to your pipeline configuration, reducing the need for multiple iterations.
The main limitation is that it can only edit the main .gitlab-ci.yml file.
3. Use gitlab-ci-local to run pipelines locally
Imagine being able to run your pipelines locally without the need to push, wait, or pollute the GitLab instance. Even better, what if you could achieve this without even interacting with the GitLab instance at all?
Initially, the gitlab-runner exec command was intended to fulfill this purpose. However, over time, the code has diverged too much from an actual pipeline running, rendering it capable of handling only 20% of the keywords. As a result, it is no longer usable. There is an official issue on this matter, but progress has been minimal.
Fortunately, the open-source community has made remarkable efforts to implement a local runner, resulting in several options:
firecow
/
gitlab-ci-local
Tired of pushing to test your .gitlab-ci.yml?
Tired of pushing to test your .gitlab-ci.yml?
Run gitlab pipelines locally as shell executor or docker executor.
Get rid of all those dev specific shell scripts and make files.
Table of contents
Installation
Linux based on Debian
Users of Debian-based distributions should prefer the the Deb822 format, installed with:
sudo wget -O /etc/apt/sources.list.d/gitlab-ci-local.sources https://gitlab-ci-local-ppa.firecow.dk/gitlab-ci-local.sources
sudo apt-get update
sudo apt-get install gitlab-ci-local
If your distribution does not support this, you can run these commands:
curl -s "https://gitlab-ci-local-ppa.firecow.dk/pubkey.gpg" | sudo apt-key add -
echo "deb https://gitlab-ci-local-ppa.firecow.dk ./" | sudo tee /etc/apt/sources.list.d/gitlab-ci-local.list
sudo apt-get update
sudo apt-get install gitlab-ci-local
Note that the path /etc/apt/sources.list.d/gitlab-ci-local.list is used in the file…
mdubourg001
/
glci
🦊 Test your Gitlab CI Pipelines changes locally using Docker.
glci 🦊
Ease GitLab CI Pipelines set-up by running your jobs locally in Docker containers.
Why ? Because I did not want to commit, push, and wait for my jobs to run on the GitLab UI to figure I forgot to install make before running make build.
📣 Disclaimer: this is a helper tool aiming to facilite the process of setting up GitLab CI Pipelines. glci does NOT aim to replace any other tool.
Installation
You need to have Docker installed and running to use glci.
yarn global add glci
Usage
At the root of your project (where your .gitlab-ci.yml is):
glci
.glci to your .gitignore file to prevent committing it.
Options
--only-jobs [jobs]
Limiting the jobs to run to the comma-separated list of jobs name given. Handy when setting up that stage-three job depending on that first-stage job artifacts.
Example:
glci --only-jobs=install,test:e2e
…
Adrian DC / gitlabci-local · GitLab
Launch .gitlab-ci.yml jobs locally - https://pypi.org/project/gitlabci-local/
gitlab.com
cunity / gitlab-emulator · GitLab
Run most gitlab jobs locally before pushing to your gitlab server. Ideal to validate your yaml changes or test simple builds out locally.
gitlab.com
Among these options, the one that has gained the most traction is gitlab-ci-local:
Here are some key features of gitlab-ci-local:
- The only prerequisite is having functional Docker commands.
- You can trigger the execution of the entire pipeline, a single job, or the fastest path to a job using the
needs:chain. - No need to push changes or establish a connection to the server. Committing is also not required.
- It supports 99% of the keywords used in GitLab CI pipelines and continuously closes the gap. This means that even complex real-life pipelines with artifacts and lightning-fast cache can be executed seamlessly.
- YAML from includes are seamlessly downloaded
- GitLab CI predefined variables are populated with valid values.
- Global, group, and project variables can be managed using files.
- It supports running unlimited jobs in parallel.
Validate pipeline YAML
To validate your pipeline, simple try to display the jobs list:
$ gitlab-ci-local --list
parsing and downloads finished in 178 ms
name description stage when allow_failure needs
📦✅-webapp-package-and-test 📦 package ✅ test on_success false
📦✅-backend-package-and-test 📦 package ✅ test on_success false
🐳🧪-webapp-build-push-dev 🐳 build-push on_success false
🐳🧪-backend-build-push-dev 🐳 build-push on_success false
🗑✨-delete-namespace-mr ☸ deploy manual true
☸🧪-webapp-deploy-dev ☸ deploy on_success false
☸🧪-backend-deploy-dev ☸ deploy on_success false
✔-e2e-tests ✔ e2e tests on_success false
It helps test your rules. For this, you can also force some predefined variables, to test alternative branching situations.
Run a full pipeline
By default gitlab-ci-local will run the full pipeline for the current branch, or for the current merge request, depending of the predefined variables you overrode. It will trigger and run jobs in parallel, as a GitLab runner would do:
$ gitlab-ci-local
Run just a job
We can also run a single job:
$ gitlab-ci-local 📦✅-webapp-package-and-test
Or a job and its dependencies before it (needs: chain and previous stages):
$ gitlab-ci-local 🐳🧪-webapp-build-push-dev --needs
📦✅-webapp-package-and-test starting node:18.11-alpine (📦 package ✅ test)
📦✅-webapp-package-and-test copied to docker volumes in 7.72 s
📦✅-webapp-package-and-test mounting cache for path projects/webapp/node_modules
📦✅-webapp-package-and-test mounting cache for path projects/webapp/.next/cache
📦✅-webapp-package-and-test $ cd projects/$MODULE
📦✅-webapp-package-and-test $ if [ ! -d node_modules/.bin ]; then yarn install; fi
[...]
Handle UI variables
Global, group and project variables can be handled in $HOME/.gitlab-ci-local/variables.yml, or locally in each project.
Official example:
$HOME/.gitlab-ci-local/variables.yml
project:
gitlab.com/test-group/test-project.git:
# Will be type Variable and only available if remote is exact match
AUTHORIZATION_PASSWORD: djwqiod910321
gitlab.com:project/test-group/test-project.git: # another syntax
AUTHORIZATION_PASSWORD: djwqiod910321
group:
gitlab.com/test-group/:
# Will be type Variable and only available for remotes that include group named 'test-group'
DOCKER_LOGIN_PASSWORD: dij3213n123n12in3
global:
# Will be type File, because value is a file path
KNOWN_HOSTS: "~/.ssh/known_hosts"
DEPLOY_ENV_SPECIFIC:
type: variable # Optional and defaults to variable
values:
"*production*": "Im production only value"
"staging": "Im staging only value"
FILE_CONTENT_IN_VALUES:
type: file
values:
"*": |
Im staging only value
I'm great for certs n' stuff
4. Declare test branches to simulate long-lived branches and tags
Now that we have the tools to test our pipeline changes with confidence before pushing them, there is still one tricky aspect to consider: how will our pipeline behave on long-lived branches after the merge request is closed?
While we can use gitlab-ci-local locally to force certain variables, such as setting $CI_COMMIT_BRANCH to main in $HOME/.gitlab-ci-local/variables.yml, this approach is only applicable locally and does not serve as valid proof for people reviewing our merge request.
A satisfying solution is to always declare special branch/tag cases and include a corresponding test version. This means that our jobs will run on both main and main-test, on tags and tags-test branch, for example:
deploy-module-staging:
extends:
- .staging
- .deploy-module
rules:
- if: $CI_COMMIT_TAG || $CI_COMMIT_BRANCH == "tag-test"
By doing this, we can test all branches conveniently from our current merge request without actually merging anything. We can create these test branches from our latest commit in the GitLab UI. Once we are satisfied with the pipeline execution (or just the composition of pipeline jobs for sensitive tasks), we can stop the associated pipeline and delete the test branch. If we are not satisfied, we can simply add more commits to our merge request and recreate (or rebase) our test branch(es). When everything seems fine in all scenarios, we can confidently proceed with merging our merge request, knowing that it won't cause any unexpected issues.
Wrapping up
By following these practices and leveraging the available tools, you can significantly reduce the time spent on iterating and testing GitLab CI pipelines. This will lead to a more efficient and enjoyable pipeline development process, ultimately improving your overall development workflow.
So go ahead and implement these tips in your GitLab CI pipeline development, and experience the benefits of faster iterations and more reliable pipelines. Happy coding!
Did this pieces of advice help you ? Do you have others on the same subject ? Please share in the comments below 🤓.
Illustrations generated locally by Automatic1111 using ZavyComics model with DieselPunkAI LoRA
Further reading
This article was enhanced with the assistance of an AI language model to ensure clarity and accuracy in the content, as English is not my native language.
Top comments (2)
Top ! Constat que je partage au quotidien (cf mon talk au devops d-day de 2023).
Je vais (re)-tester certains de ces outils.
Je pourrais rajouter utiliser un moteur de CI indépendant de votre CI. Les options :
a) tout faire avec make et docker
b) dagger
c) earthly
Le problème avec l'outillage intermédiaire supplémentaire, c'est l'effet boîte noire pour les développeurs, qui viennent dire ensuite "Ta CI ne fonctionne pas" (pas de ownership).
Je préfère considérer que Maven ou NPM sont déjà des makes ! Et la CI est jouable en local avec gitlab-ci-local, comme un make.
Les autres sont aussi des abstractions supplémentaires.
On rentre dans le subjectif là je pense 😉