Jenkins Pipelines and their dirty secrets 3.

Tips and Tricks of Pipelines

How did this chapter come to be? I was basically just pasting here stuff during work. There are many subtleties to pipelines, I recommend to all of you to create an own Jenkins as a playground of study, and experiment with it.

---

Handling build dependencies

Now before we start, let's keep in mind that pipeline scripts can be formulated in either Declarative or Scripted syntax. The first example code in this section is formulated in Declarative Pipeline syntax. If you don't know what the difference is, read up the first chapter. Let's see the following problems:

1. You have two jobs following each other:

#!/usr/bin/env groovy

// HEADS UP: This is Declarative Pipeline syntax

pipeline {
agent any
    stages {
        stage("build") {
            steps {
                build('job-1')
                build('job-2')
            }
        }
    }
}

In this case, the first build step will be executed first, but the whole job will stop and fail if the first job fails. But let's say that you don't want this. Instead, you want the following:

• You want both of them to run, even if the first fails, and only fail if the second job fails
• You want the jobs to run after each other but not parallel.

Here's your solution: Use the propagate attribute, and set it to false. If disabled, then this step succeeds even if the downstream build is unstable, failed.

build(job: 'job-1', propagate: false)
// This step will run even if job-1 failed
build(job: 'job-2')

This case the job will start the second build step even if the first one failed. But there is a catch. If the first one failed, you will not know this from the pipeline's result. The pipeline job's result will evaluate to be a SUCCESS even if the first job fails. This is not necessarily a problem, but the third example will offer a solution for this issue.

2. You want to run parallel builds:

#!/usr/bin/env groovy

pipeline {
    agent any
    stages {
        stage("test") {
            steps {
                parallel (
                    "Unit Test" : {
                        build("unit-test-job")
                    },
                    "Component Test" : {
                        build("component-test-job")
                    },
                    "Build" : {
                        build("build-job")
                    }
                )
            }
        }
    }
}

Yes but is there a catch? Yes there is:
For one, you can't have anything else, but that parallel block in that stage. If you would add a separate build step, like this:

//    !!!!!!!!!!!!!!!!!!!!!!!!!!
//    DONT USE! BAD CODE

            steps {
                parallel (
                    "Unit Test" : {
                        build("unit-test-job")
                    },
                    "Component Test" : {
                        build("component-test-job")
                    },
                    "Build" : {
                        build("build-job")
                    }
                )
                // extra build step together with parallel block
                build("extra-job")
            }
//    !!!!!!!!!!!!!!!!!!!!!!!!!!

...you would get the following ERROR:
WorkflowScript: 6: Invalid step "parallel" used - not allowed in this context - The parallel step can only be used as the only top-level step in a stages step block.

A quick solution is to reformulate your script in Scripted pipeline syntax:

node {
// HEADS UP: this is Scripted Pipeline syntax
    stage("Build") {
        parallel (
            "First Build" : {
                build("first-build-job")
            },
            "Second Build" : {
                build("second-build-job")
            }
        )
        build("test-job")
        parallel (
            "Third Build" : {
                build("third -build-job")
            },
            "Last Build" : {
                build("last-build-job")
            }
        )
    }
}

3. You want to examine the build

The build step has a hidden attribute: wait (just like propagate, which you have seen before in this chapter).

build(job: 'example-job', wait: true)

WARNING: I've seen online that many people confuse wait with propagate, when they want to ignore the failure of a job in the pipeline. Let's see the differences:

build(job: 'example-job', wait: false): This means that the pipeline will not wait for the result of this build step, just starts it and jumps to the next step. If you put blocks like this after each other, they will be started one after another, without waiting for each other to finish (Actually almost like a parallel block).
build(job: 'example-job', propagate: false): This means that the pipeline will first execute this step, and continue to the next step even if this step failed.

wait is true as default, so normally you don't have to add it to the code. In case you don't turn it false, the return value of the build step will be an object on which you can obtain read-only properties: so you can inspect its .result and so on.

node {
    stage("example") {
        echo build("example-job").result
    }   

With this, you can actually achieve to set the build's result even if you ignored failing builds with setting propagate: false: Basically what you have to do is just to set the jobs result by hand after examining given conditions with an if block.
This way the pipeline will keep running even if the first job fails, but you will see from the result of the pipeline if something was wrong.

node {
    stage("example") {
        b = build(job: "example-job", propagate: false).result
        if(b == 'FAILURE') {
            echo "First job failed"
            currentBuild.result = 'UNSTABLE' // of FAILURE
        }
    }
    stage("test") {
        build("test-job")
    }
}

Comments

[YangFeng]

Hi, Pencillr. Thanks for sharing very useful tips about pipeline in your article. Do you have any idea about getting all names of stage before they have been executed? I mean, how to get all the defined stages' name in the beginning of Jenkinfile?

[Richard Lenkovits]

Hi, thank you. I don't know of any way that would get you the name of the stages before the stages run, in the Jenkinsfile. Although you could do several things to have the stage names beforehand.
First, you could instantiate a list with the names of the stages at the beginning, and you could reference the elements of it at the stage declaration.

[YangFeng]

Thanks for your reply. I think I understand what you mean, a parameters{} block or a input method is feasible.

[mdali_11]

Is there a way to check the whether Build A is running or not, If Build A running
Build B should be wait until this Build A has to complete, and then Build B has to complete its build, the pipeline to check the Build A is in Build B.....

[vamsi krishna]

Hi Pencillr, Thanks alot for the post. I have one requirement where I have to trigger 2nd stage in pipeline at different time and date after the initial stage. Can you please help me if you have any suggestions?

[Benjamin Paz]

Did you ever figure out how to trigger 2nd stage in pipeline?

I also have a requirement to trigger just a stage in Jenkinsfile pipeline. This stage should only run daily but all other stages triggered by a code push. Any ideas?

[Paweł Kowalski]

Have the same case, no luck so far.

[Casey Vega]

Use what is known as a parameterized pipeline. You can set default environment variables for code runs.

when you want to change the behavior of the pipeline stage you can modify the parameter. If you set the toggle parameter to false in the UI, or by calling the build via URL, it will now skip the first stage because it's not true

Calling via URL:
http://server/job/project/buildWithParameters?toggle=false

pipeline {
  agent any
  parameters {
    booleanParam(name: 'toggle', defaultValue: true)
  }
  stages {
    stage('optional stage') {
      when (toggle: true) {
        sh `make clean`
      }
    }
  }
}

[Danis Nya]

Hi, Pencillr!
Thanks a lot for your tips, it is really useful. I have two question:

1. When i execute my stages in parallel, job do not stopped when 1 of 2 stages is fail. How can i fix it?
2. How can i sorted my logs in parallel execution?

Best regards.

[Lewis Evans]

for #1. you can do

parallel (
    "Unit Test" : {
        build("unit-test-job")
    },
    "Component Test" : {
        build("component-test-job")
    },
    "Build" : {
        build("build-job")
    },
    failFast: true
)

for #2. Not sure you can do that unless you run sequentially

[Monika1312]

Hi Pencillr, Thanks for this post.
I have one question.
I understood the use of propagate: false but how to show the stage as fail/red when something is actually failed. My job is failed and shows as green and I don't want to go and click every time inside the stage result.

[Benjamin Paz]

Hi Pencillr, I would like to trigger just a pipeline stage in Jenkinsfile. This stage should only run daily but all other stages triggered by a code push. Any ideas?

[Gustavo]

Great and very useful tips.
Thank you, Richard!!!