loading...
Cover image for Jenkins Pipelines and their dirty secrets 3.

Jenkins Pipelines and their dirty secrets 3.

pencillr profile image Richard Lenkovits Updated on ・4 min read

Tips and Tricks of Pipelines

How did this chapter come to be? I was basically just pasting here stuff during work. There are many subtleties to pipelines, I recommend to all of you to create an own Jenkins as a playground of study, and experiment with it.


Handling build dependencies

Now before we start, let's keep in mind that pipeline scripts can be formulated in either Declarative or Scripted syntax. The first example code in this section is formulated in Declarative Pipeline syntax. If you don't know what the difference is, read up the first chapter. Let's see the following problems:

1. You have two jobs following each other:

#!/usr/bin/env groovy

// HEADS UP: This is Declarative Pipeline syntax

pipeline {
agent any
    stages {
        stage("build") {
            steps {
                build('job-1')
                build('job-2')
            }
        }
    }
}

In this case, the first build step will be executed first, but the whole job will stop and fail if the first job fails. But let's say that you don't want this. Instead, you want the following:

  • You want both of them to run, even if the first fails, and only fail if the second job fails
  • You want the jobs to run after each other but not parallel.

Here's your solution: Use the propagate attribute, and set it to false. If disabled, then this step succeeds even if the downstream build is unstable, failed.

build(job: 'job-1', propagate: false)
// This step will run even if job-1 failed
build(job: 'job-2')

This case the job will start the second build step even if the first one failed. But there is a catch. If the first one failed, you will not know this from the pipeline's result. The pipeline job's result will evaluate to be a SUCCESS even if the first job fails. This is not necessarily a problem, but the third example will offer a solution for this issue.

2. You want to run parallel builds:

#!/usr/bin/env groovy

pipeline {
    agent any
    stages {
        stage("test") {
            steps {
                parallel (
                    "Unit Test" : {
                        build("unit-test-job")
                    },
                    "Component Test" : {
                        build("component-test-job")
                    },
                    "Build" : {
                        build("build-job")
                    }
                )
            }
        }
    }
}

Yes but is there a catch? Yes there is:
For one, you can't have anything else, but that parallel block in that stage. If you would add a separate build step, like this:

//    !!!!!!!!!!!!!!!!!!!!!!!!!!
//    DONT USE! BAD CODE

            steps {
                parallel (
                    "Unit Test" : {
                        build("unit-test-job")
                    },
                    "Component Test" : {
                        build("component-test-job")
                    },
                    "Build" : {
                        build("build-job")
                    }
                )
                // extra build step together with parallel block
                build("extra-job")
            }
//    !!!!!!!!!!!!!!!!!!!!!!!!!!

...you would get the following ERROR:
WorkflowScript: 6: Invalid step "parallel" used - not allowed in this context - The parallel step can only be used as the only top-level step in a stages step block.

A quick solution is to reformulate your script in Scripted pipeline syntax:

node {
// HEADS UP: this is Scripted Pipeline syntax
    stage("Build") {
        parallel (
            "First Build" : {
                build("first-build-job")
            },
            "Second Build" : {
                build("second-build-job")
            }
        )
        build("test-job")
        parallel (
            "Third Build" : {
                build("third -build-job")
            },
            "Last Build" : {
                build("last-build-job")
            }
        )
    }
}

3. You want to examine the build

The build step has a hidden attribute: wait (just like propagate, which you have seen before in this chapter).

build(job: 'example-job', wait: true)

WARNING: I've seen online that many people confuse wait with propagate, when they want to ignore the failure of a job in the pipeline. Let's see the differences:

build(job: 'example-job', wait: false): This means that the pipeline will not wait for the result of this build step, just starts it and jumps to the next step. If you put blocks like this after each other, they will be started one after another, without waiting for each other to finish (Actually almost like a parallel block).
build(job: 'example-job', propagate: false): This means that the pipeline will first execute this step, and continue to the next step even if this step failed.

wait is true as default, so normally you don't have to add it to the code. In case you don't turn it false, the return value of the build step will be an object on which you can obtain read-only properties: so you can inspect its .result and so on.

node {
    stage("example") {
        echo build("example-job").result
    }   

With this, you can actually achieve to set the build's result even if you ignored failing builds with setting propagate: false: Basically what you have to do is just to set the jobs result by hand after examining given conditions with an if block.
This way the pipeline will keep running even if the first job fails, but you will see from the result of the pipeline if something was wrong.

node {
    stage("example") {
        b = build(job: "example-job", propagate: false).result
        if(b == 'FAILURE') {
            echo "First job failed"
            currentBuild.result = 'UNSTABLE' // of FAILURE
        }
    }
    stage("test") {
        build("test-job")
    }
}

Discussion

pic
Editor guide
Collapse
yooofeng profile image
YangFeng

Hi, Pencillr. Thanks for sharing very useful tips about pipeline in your article. Do you have any idea about getting all names of stage before they have been executed? I mean, how to get all the defined stages' name in the beginning of Jenkinfile?

Collapse
pencillr profile image
Richard Lenkovits Author

Hi, thank you. I don't know of any way that would get you the name of the stages before the stages run, in the Jenkinsfile. Although you could do several things to have the stage names beforehand.
First, you could instantiate a list with the names of the stages at the beginning, and you could reference the elements of it at the stage declaration.

Collapse
yooofeng profile image
YangFeng

Thanks for your reply. I think I understand what you mean, a parameters{} block or a input method is feasible.

Collapse
forpix profile image
mdali_11

Is there a way to check the whether Build A is running or not, If Build A running
Build B should be wait until this Build A has to complete, and then Build B has to complete its build, the pipeline to check the Build A is in Build B.....

Collapse
vamsichow profile image
vamsi krishna

Hi Pencillr, Thanks alot for the post. I have one requirement where I have to trigger 2nd stage in pipeline at different time and date after the initial stage. Can you please help me if you have any suggestions?

Collapse
bendavidpaz profile image
Benjamin Paz

Did you ever figure out how to trigger 2nd stage in pipeline?

I also have a requirement to trigger just a stage in Jenkinsfile pipeline. This stage should only run daily but all other stages triggered by a code push. Any ideas?

Collapse
pavelloz profile image
Paweł Kowalski

Have the same case, no luck so far.

Thread Thread
cvega profile image
Casey Vega

Use what is known as a parameterized pipeline. You can set default environment variables for code runs.

when you want to change the behavior of the pipeline stage you can modify the parameter. If you set the toggle parameter to false in the UI, or by calling the build via URL, it will now skip the first stage because it's not true

Calling via URL:
http://server/job/project/buildWithParameters?toggle=false

pipeline {
  agent any
  parameters {
    booleanParam(name: 'toggle', defaultValue: true)
  }
  stages {
    stage('optional stage') {
      when (toggle: true) {
        sh `make clean`
      }
    }
  }
}
Collapse
danis9yearsago profile image
Danis Nya

Hi, Pencillr!
Thanks a lot for your tips, it is really useful. I have two question:

  1. When i execute my stages in parallel, job do not stopped when 1 of 2 stages is fail. How can i fix it?
  2. How can i sorted my logs in parallel execution?

Best regards.

Collapse
lewisevans profile image
Lewis Evans

for #1. you can do

parallel (
    "Unit Test" : {
        build("unit-test-job")
    },
    "Component Test" : {
        build("component-test-job")
    },
    "Build" : {
        build("build-job")
    },
    failFast: true
)

for #2. Not sure you can do that unless you run sequentially

Collapse
monika1312 profile image
Monika1312

Hi Pencillr, Thanks for this post.
I have one question.
I understood the use of propagate: false but how to show the stage as fail/red when something is actually failed. My job is failed and shows as green and I don't want to go and click every time inside the stage result.

Collapse
bendavidpaz profile image
Benjamin Paz

Hi Pencillr, I would like to trigger just a pipeline stage in Jenkinsfile. This stage should only run daily but all other stages triggered by a code push. Any ideas?

Collapse
gusgonnet profile image
Gustavo

Great and very useful tips.
Thank you, Richard!!!