DEV Community

Cover image for GitLab CI: Creating your own pipeline template library 👷

GitLab CI: Creating your own pipeline template library 👷

Minh Trinh on May 19, 2020

As developers, we all know that it is a good idea to reuse code as much as possible. We all know the DRY mantra - Don't Repeat Yourself. Functions,...
Read full post
Collapse
 
gregorip02 profile image
Gregori Piñeres •

Hi, thanks for sharing!

What if I want to re-use a series of scripts in my jobs? I know you could write separate scripts to files e.g. scripts/deploy.sh and call its execution whenever you need it in the .gitlab-ci.yml file.

The extends: .my-reusable-job sentence could be overwritten.

# .gitlab-ci.yml

.my-reusable-job:
  script:
    - echo "Hello from .my-reusable-job"

testing:
  extends: .my-reusable-job
  script:
    - echo "Hello from testing job"
Enter fullscreen mode Exit fullscreen mode

Only prints "Hello from testing job". My specific question is if I can use extends without overwriting it later?

Collapse
 
anhtm profile image
Minh Trinh • • Edited

Hi Gregori! Thanks for posting your question :)

I think this is an intended feature for extends. If you want the reusable script to not be overwritten, there's 2 ways I can think of:

  1. Use before_script or after_script for reusable commands so that they're appended before or after the new script: 
.my-reusable-job:
  before_script:
    - echo "First, hello world"

testing:
  extends: .my-reusable-job
  script:
    - echo "Second, hello world, again!"
Enter fullscreen mode Exit fullscreen mode

This thus prints "First, hello world" and "Second, hello world, again!"

  1. Use YAML anchors to specify reusable scripts: 
# &my_reusable_commands is the name of the anchor
.my_reusable_commands: &my_reusable_commands
  - echo 'First, hello world'

testing:
  script:
   # refer to the above anchor with *<anchor_name>
    - *my_reusable_commands
    - echo "Second, hello world, again!"
Enter fullscreen mode Exit fullscreen mode

This will print the same 2 lines just like the first approach.

Let me know if you have any questions!

Collapse
 
gregorip02 profile image
Gregori Piñeres •

Thankssssss, it works 🙏🏾🙏🏾🙏🏾🙏🏾

Thread Thread
 
anhtm profile image
Minh Trinh •

Awesome! 🎉 I'm happy to help :)

Collapse
 
tobiashochguertel profile image
Tobias Hochgürtel •

Hey Minh, that’s a well written article about GitLab-Ci templates functionality, and the first which is valuable. Why? because it just shows what is hidden by this GitLab term ci template, it’s like that you can search for this term and you end opening tabs over tabs because you find articles which all talk about this feature - but in a way which is so useless that you just learn „yes that is what I need now, they can do it, it’s existing...“ but they all forget the main point, you want to see how to implement / use it / done real work with it.

Please write more - I like your way, to show me „ah okay, we have a Feature...“, „... and here you use it like this“, „do that..“ and you get it explained in one article without thousand additional links which you have to open to see a snippet, additional documentations on GitLab.com which all show you nothing usefully, and in the end you read a lot but you have no clue how to use this feature.

You got it done, in a way I like it very much - to write it down, on just one paper A4/letter which shows me how to implement and work with it.

So thanks, and how I can inspire you to write more articles, because you are do it great.

Collapse
 
anhtm profile image
Minh Trinh •

Thank you Tobias for the kind words! You definitely inspired me to write more :D

I agree with you that articles around CICD is usually limited to being generic & theoretical. The reason, I think, is because it requires the author to have real hands-on experience on the tech (ie: gitlab ci/jenkins/etc) to really understand how to apply it in the real world. There are many implicit components to building a pipeline that only by reading the docs certainly isn't enough.

I'll try to write more "how to..." articles like this one in the future and hope to share my own insights on the subject (which I think is crucial to make an article useful)

If possible, please let me know what topic you'd like to read more from. Thanks!

Collapse
 
tobiashochguertel profile image
Tobias Hochgürtel •

I have few or one idea, as a follow up article, you could show how to make a a pipeline more generic, for example I have a pipeline for python which is also possible to do specific python projects like django, because there isn't to much difference. I use there often one-liner shell script steps like:

- "[ -f .env.ci ] && cp .env.ci .env"
- "[ -f manage.py ] && python manage.py test"
Enter fullscreen mode Exit fullscreen mode

and then also how a reusable template can be used look, and used. For example:

my Project pipeline looks like this:

include:
  - project: 'infrastructure-hochuertel.work/gitlab-ci.yml'
    ref: master
    file: '/python/Python-Projects.gitlab-ci.yml'

variables:
  PROJECT_DIRECTORY: 'mysite'
Enter fullscreen mode Exit fullscreen mode

I have only to overwrite one variable to select the correct directory which is to be used to test. That I have to set the variable has a repository structure semantic, I could also have all python project in the same directory of a new repository, but I do it like this. -because I will use the same template again for the same repository but with a different directory (to release python packages to pypi, we refactored some django-apps into python packages to reuse them).

the interesting thing which I have in mind is maybe this:
Another topic which is often not really clear is how to versioning with a pipeline, how can we trigger a release (manual by API, or by a specific trigger like changed file) in the way that we have not to do a change in the gitlab-ci.yml file (to activate release button, and then deactivate release button)...
Releasing and Versioning is often confusing topic when I look into development teams, mostly they have per project only one package which they "release" what they do is just deploy.

hmm, I could write more and more ideas, as I sit here and write down what I have in mind about what you could write, often topics which I have to cover myself, also.

  • Multi Pipeline Triggering,
  • How to release a new Documentation Version next to the already existing version of the documentation (think like the Java API documentation for each release), how could this be organized and done, maybe via gitlab pages feature, or different.
Collapse
 
avik735 profile image
Avi Bis •

I created a profile just to comment and ask some questions. First of all, thank you for such a well written guide. There are very few articles on GitLab CI on the internet. Most are utterly theoretical and very few provides actual workable examples.

Coming to my question, I followed this guide to create a template repository. I have some job templates in the repo, and a pipeline config template which includes these locally available job templates. I am using the pipeline config as a template in other projects. One of the job templates is a maven build job, which looks like this:

.compile_commit:
  variables:
    POM: 'pom.xml'
  only:
    - branches
  tags:
    - ac-billing
  image: $IMAGE_PREFIX/maven:$MAVEN_TAG
  script:
    - mvn -f $POM $MAVEN_CLI_OPTS clean install $MAVEN_OPTS
  artifacts:
    paths:
      - target/*.zip
      - .m2/repository
    expire_in: 1 hour
Enter fullscreen mode Exit fullscreen mode

In the pipeline config template, I have extended this job template as

include: '/job_templates/compile_commit.gitlab-ci.yml'
compile_commit:
  stage: build
  extends: .compile_commit
  variable:
    POM: pom.xml
Enter fullscreen mode Exit fullscreen mode

Finally, in the actual project (project A), I have included the pipeline config template as

include:
  - project: 'mygroup/ci_templates'
    ref: 'master'
    file: '/job_templates/pipeline.gitlab-ci.yml'

variables:
  POM: 'myfolder/pom.xml'
Enter fullscreen mode Exit fullscreen mode

However, when I run the pipeline in project A, the build process fails to find the pom.xml path and the build fails. What am I doing wrong?

Collapse
 
anhtm profile image
Minh Trinh •

Hi Avi, thanks for stopping by! It's amazing to see people like you finding the guide useful and actually following it.

I think the problem is that in your project A, you're declaring the variable POM: 'myfolder/pom.xml' on the pipeline level, and not on the job level. Hence it's perceived as a different variable from the original .compile_commit::variables::POM and GitLab does not parse it into the job

The easiest way to solve this is to make the template config job to be hidden, and then extend it again in Project A and change the variable POM:

In template config:

include: '/job_templates/compile_commit.gitlab-ci.yml'
.compile_commit_config: # make this job hidden
  stage: build
  extends: .compile_commit
  variable:
    POM: pom.xml
Enter fullscreen mode Exit fullscreen mode

In Project A:

include:
  - project: 'mygroup/ci_templates'
    ref: 'master'
    file: '/job_templates/pipeline.gitlab-ci.yml'

compile_commit:
  extends: .compile_commit_config
  variables:
    POM: 'myfolder/pom.xml'  # add your variable inside the new job
Enter fullscreen mode Exit fullscreen mode

Another approach would be to set the POM variable to be at the pipeline level from the beginning (remove it from the job and add it in the pipeline level). Then, override it in your Project A like you did. However, I haven't tested this solution so I can't 100% tell you that it will work. But it's worth trying it out. Let me know if that works also!

Minh

Collapse
 
himito profile image
Jaime Arias Almeida •

Thank you for sharing this !!! Very useful !

Collapse
 
anhtm profile image
Minh Trinh •

Thanks for reading! :)