Jenkins Pipelines - surely this is a bug??

EDIT -- the mystery has been solved.

In my last post, I talked about a self-calling pipeline, and gave an example of a dispatcher which seemed to work when calling it against my single-node label. I mentioned the frustration of newly learning technologies.

Welp. That anger is back. I nearly gave up on the pipeline altogether to commit to another week (on the face of it, probably just 2 days, but you know how estimations work, right?) of ripping up my work and starting the dispatcher again.

Seriously, Jenkins should come with a health warning.

If I get around to starting a brand new pipeline system ever again, I will not be choosing Jenkins before seriously trying out the competition. On the flip side, I hope Jenkins is not the best of a bad lot, because that would seriously suck gobstoppers

We use Jenkins as part of our hardware testing system as I probably mentioned before, and the need to start a same pipeline in parallel comes up a fair bit. In developing such a pipeline, we hit against an odd behaviour: even as the function that built up a list of job calls seemed to dynamically iterate over a list of node names as one would expect, by the time parallel was called on the returned map object, every instance of the build definition was set to the same value, namely, the last value the for loop obtained during iteration!

prepared_node_tests = [:]

def prepare_all_nodes_for_test(agent_label) {
    def nodelist = nodesByLabel(label: "${agent_label}")

    for (agentName in nodelist) {
        println "Preparing task for " + agentName

        prepared_node_tests[ agentName ] = {
            build job: 'Single_build_job',
            parameters: [
                string(name: 'TEST_AGENT_LABEL', value: agentName),
            ]
        }
    }
}


pipeline {
    agent none

    parameters {
        string(name: 'TEST_AGENT_LABEL', defaultValue: 'agent_collection',
          description: 'Run pipeline against all nodes with this name/label')
    }

    stages {
        stage('Kick-off agents') {
            steps {
                script {
                    prepare_all_nodes_for_test("${params.TEST_AGENT_LABEL}")
                    parallel prepared_node_tests
                }
            }
        }
    }
}

In the above example, the parameter line where we set string(name: 'TEST_AGENT_LABEL', value: "${agentName}") looks like it should interpolate immediately, and resolve with the value being set as a fixed string.

Instead, each job gets effectively called against the same agent - the last agent value that was seen in the for loop. It is as if the string was not interpolated during assignment in prepare_all_nodes_for_test(), but instead at execution time. Which should not happen because agentName should by then be completely out of scope.

The solution was to force the name resolution earlier, before assigning into the map. The reason this works is completely lost on me, but it seems that confining the object definition in its own function, and returning that out directly, causes the interpolation to happen.

prepared_node_tests = [:]

def get_one_build_object(name) {
    return {
        build job: 'Single_build_job',
        parameters: [
            string(name: 'TEST_AGENT_LABEL', value: name),
        ]
    }
}

def prepare_all_nodes_for_test(agent_label) {
    def nodelist = nodesByLabel(label: "${agent_label}")

    for (agentName in nodelist) {
        println "Preparing task for " + agentName

        prepared_node_tests[ agentName ] = get_one_build_object(agentName)
    }
}


pipeline {
    agent none

    parameters {
        string(name: 'TEST_AGENT_LABEL', defaultValue: 'agent_collection',
          description: 'Run pipeline against all nodes with this name/label')
    }

    stages {
        stage('Kick-off agents') {
            steps {
                script {
                    prepare_all_nodes_for_test(params.TEST_AGENT_LABEL)
                    parallel prepared_node_tests
                }
            }
        }
    }
}

Whether this is some obscure Groovy bug, or whether Jenkins has done something very funky to the Groovy engine to implement declarative pipelines remains a mystery to me, for now...

But it does continue to make Jenkins feel like it is royally fuzzed up in ways that defy imagination...