Gitlab CI/CD Multi-Project Pipeline

Gitlab CI/CD

Gitlab CI/CD is the continuous integration and continuous delivery solution created by Gitlab, if you need help understanding the basics please refer to other sources like:

• Gitlab
• Getting started with GitLab CI/CD - GitLab Unfiltered
• Gitlab CI pipeline tutorial for beginners - Valentin Despa

Multi-Project Pipeline

Is a feature that allows us to trigger pipelines that are hosted in other projects.

This is especially useful for microservices projects that need a way to deploy everything in the correct order, for projects that need to run E2E tests before deploying to production, or any other case that you can think of that needs integrated pipelines between projects.

Launching another's project pipeline was initially introduced to Gitlab in the form of API Triggers. Triggers are excellent when you need to launch a pipeline outside of Gitlab or for another project that you or your organization do not own. But the use of this type of triggers needs additional configurations like create a trigger token or use curl inside the job script.

Multi-Project Pipeline became free for all users in early 2020, but without the visualization of the pipeline triggered, like this:

In this case, we are going to explain some of the configurations in the Multi-Project Pipeline feature and from a free account ;)

Use Case:

Let's assume the following:

• We are developing an awesome app by our development team.
• The development team has already done some functionality of the awesome app and is already deployed to production.
• But we also have a QA team that needs to make E2E testing to our awesome app.

Projects

• my-awesome-project: This is the project with the code we want to deploy to production.

• e2e-testing-qa: This is the project we want to trigger before the deployment phase in my-awesome-project, the main purpose of this project is to do E2E testing in a staging environment.

* This projects reside inside a group in Gitlab called my-awesome-group-trigger

Pipelines

Current pipeline flow:

In the current flow we have a stage to unit-testing, build our Docker image to later be deployed to production. We need to find a way to integrate this flow of the development team with the QA team and add some E2E tests.

First desired pipeline flow:

So this is a problem because the code related to our E2E tests would be in the same project that our production code, and this involves the two teams. So probably the best approach, in this case, is to have separate projects, this means we need to use a strategy to launch and sync our deployment pipelines.

Final pipeline flow:

This would integrate the two projects and only when all the previous stages to deploy are successfully passed the code would be launched to production.

* I would explain later what this Bridge job is.

CI/CD without the integration

• my-awesome-project/.gitlab-ci.yml

image: alpine

stages:
    - test
    - build
    - deploy

test:
    stage: test
    script: 
        - echo "This is the test job."

build:
    stage: build
    script: 
        - echo "This is the build job."

deploy:
    stage: deploy
    script: 
        - echo "This is the deploy job."

• e2e-testing-qa/.gitlab-ci.yml

image: alpine

stages:
    - deploy
    - e2e

deploy:
    stage: deploy
    script: 
        - echo "This is the deploy job."

e2e:
    stage: e2e
    script:         
        - echo "This is the E2E job."

As you can see we just need to make the QA integration, for this we are going to use the Multi-Project pipeline.

Bridge Job:

This is a job that has the special keyword trigger and this automatically turns this job in a bridge job, this means that its only purpose is to launch a pipeline in another project or the same project (see parent-child-pipelines).

Integration

In this particular case we only need to modify the CI/CD file in my-awesome-project, let's add the bridge stage and job, between build and deploy to make our E2E testing:

# my-awesome-project/.gitlab-ci.yml

image: alpine

stages:
    - test
    - build
    - bridge
    - deploy

test:
    stage: test
    script: 
        - echo "This is the test job."

build:
    stage: build
    script: 
        - echo "This is the build job."

bridge:
    stage: bridge
    trigger: my-awesome-group-trigger/e2e-testing-qa

deploy:
    stage: deploy
    script: 
        - echo "This is the deploy job."

Let's see how our pipelines are working together (for the following images the top pipeline is the project my-awesome-project and the bottom is e2e-testing-qa)

• Passing pipeline

• Falling pipeline

Works! But it seems like the pipeline in the project my-awesome-project it's not waiting for the e2e-testing-qa to finish, and this is not our desired behavior for this case, remember we need to make the E2E tests before the production deployment. So we need to modify our bridge job one more time to add the strategy keyword:

# my-awesome-project/.gitlab-ci.yml

...

bridge:
    stage: bridge
    trigger: my-awesome-group-trigger/e2e-testing-qa
    strategy: depend

...

• Passing pipeline

• Falling pipeline

Finally, when the E2E testing has passed in the e2e-testing-qa our deploy to production can be made automatically, and when it fails will also be falling our main pipeline.

Comments

[John Coghlan]

Hey, I work on the developer evangelism team at GitLab. Do you have any public projects using multi-project pipelines? Would love to share with our product team.