Oleksandr Kuryzhev

Posted on Oct 23 Originally published at kuryzhev.cloud

Jenkins Shared Library Structure for Reusable Pipelines

#devops #cicd

Originally published on kuryzhev.cloud

Twenty repositories, twenty Jenkinsfiles, and one security fix to the Docker build stage that now needs twenty pull requests. That is the moment most teams reach for a Jenkins shared library, and it is also the moment they tend to build the wrong one. The mechanism is simple, but the structure you choose decides whether the library stays maintainable or becomes a second, harder-to-debug copy of the same sprawl.

This piece walks through what the library mechanism does, the patterns that cause trouble, a layout that holds up, and the performance traits to keep in mind. Check the official shared libraries documentation for details that vary by plugin version.

What this actually does: a Jenkins shared library under the hood

A shared library is a Git repository that Jenkins checks out at the start of a build and adds to the Groovy classpath of that pipeline run. Your Jenkinsfile requests it with @Library('name@version') _, and Jenkins resolves the name against libraries configured globally, on a folder, or on a multibranch project.

The repository has a fixed convention, and each directory behaves differently:

vars/ holds global variables and custom steps. A file named buildImage.groovy with a call() method becomes the step buildImage(...). An optional buildImage.txt file supplies the help text shown in Pipeline Syntax. It is rendered through the configured markup formatter, but the .txt extension is required.
src/ holds regular Groovy classes in packages, like a standard Java source tree. You import them explicitly.
resources/ holds non-Groovy files (shell scripts, templates, JSON) that you load with libraryResource.
test/ is not special to Jenkins, but it is where unit tests belong by convention.

The key mental model: library code runs inside the Pipeline engine, so it is subject to the same CPS (continuation-passing style) transformation as a Jenkinsfile. That is why it can pause on sh or input and survive a controller restart, and also why some ordinary Groovy idioms misbehave.

How people use it wrong

The most common failure is treating the library as a dumping ground. Someone copies the full Jenkinsfile into vars/standardPipeline.groovy, hardcodes the stage list, and adds a boolean parameter every time a team needs an exception. Over time the step accumulates dozens of arguments, and nobody can predict what a given combination does.

Other documented pitfalls follow a pattern:

Floating versions. Loading @Library('shared') with no version, or pinning to main, means every merge to the library changes every pipeline immediately. A bad commit becomes an instance-wide outage of builds.
Logic in the wrong place. Heavy parsing, loops over large collections, or HTTP calls in Groovy run on the controller, not the agent. Watch out for this: CPS code competes for controller resources across all running builds.
Global state in vars. Fields kept in a vars script are easy to misuse, and CPS serialization can fail on non-serializable objects held across steps. The typical symptom is a NotSerializableException after a step that pauses.
Secrets inside the library. Tokens in the repo or in resource files are readable by anyone with read access to that repository. Use Jenkins credentials and bind them at runtime.
No tests. Library changes go straight to production pipelines, and the first test is somebody's failed release.

One more trap concerns trust. Libraries configured globally run outside the Groovy sandbox and are implicitly trusted, while folder-level libraries run sandboxed. Putting a library in the global scope for convenience widens who can effectively run unsandboxed code through a pull request. Verify who can merge to that repository.

The correct approach

Keep the Jenkinsfile as the declaration of intent, and keep the library as a set of small, composable steps. Teams call steps with a map of options; the step owns the how. Prefer many narrow steps (buildImage, scanImage, deployHelm) over one giant template, and add an opinionated template only on top of them.

A layout that scales:

jenkins-shared-lib/
├── vars/
│   ├── buildImage.groovy        # step: docker build + push
│   ├── buildImage.txt           # help text shown in Pipeline Syntax (.txt required)
│   ├── scanImage.groovy
│   └── standardService.groovy   # optional template composed from the steps above
├── src/
│   └── org/example/ci/
│       ├── ImageRef.groovy      # pure-logic helpers, easy to unit test
│       └── Validate.groovy
├── resources/
│   └── org/example/ci/
│       └── trivy-args.txt       # loaded via libraryResource
└── test/
    └── groovy/                  # unit tests for src/ and vars/

Here is a custom step that validates its input, keeps work on the agent through sh, and holds no state between calls:

// vars/buildImage.groovy
import org.example.ci.ImageRef

def call(Map cfg = [:]) {
    // fail fast with a clear message instead of a stack trace later
    if (!cfg.name) { error 'buildImage: "name" is required' }

    String tag = cfg.tag ?: env.GIT_COMMIT?.take(12) ?: 'dev'
    String ref = new ImageRef(cfg.registry ?: 'registry.example.com', cfg.name, tag).toString()

    // heavy lifting happens on the agent, not in Groovy
    sh "docker build -t ${ref} ${cfg.context ?: '.'}"
    if (cfg.push != false) {
        sh "docker push ${ref}"
    }
    return ref   // return values let callers chain steps
}

And the consuming Jenkinsfile, pinned to an immutable tag:

// Jenkinsfile
@Library('shared-ci@v2.4.0') _   // pin to a tag or commit, never a moving branch

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        script {
          env.IMAGE = buildImage(name: 'payments-api', context: 'service/')
        }
      }
    }
    stage('Scan') {
      steps { scanImage(image: env.IMAGE) }
    }
  }
}

Release the library with Git tags and semantic versioning. A breaking change to a step signature is a major version, and teams migrate on their own schedule.

Advanced patterns

Template steps with a closure body. For teams that want a near-zero Jenkinsfile, a template step can define the whole declarative pipeline and accept overrides. Keep the override surface deliberately small: a map of options and optional closures for extension points such as preBuild. If you find yourself adding a flag per team, split the template instead.

Unit-testing the library. Put pure logic into src/ classes with no Jenkins dependencies and test them with plain JUnit or Spock. For steps in vars/, the community JenkinsPipelineUnit framework can mock pipeline steps such as sh and error. Its compatibility with your Jenkins and Groovy versions is something to verify before building a test suite around it.

Testing a library change before release. A pipeline can load a branch explicitly with @Library('shared-ci@feature/new-scan') _, or use the library step to load a version dynamically. Both work only when "Allow default version to be overridden" is enabled in the library configuration. With that option on, a test job can point at a branch without touching the shared default. Watch out for the reverse risk: if overrides are allowed broadly, a Jenkinsfile author can load any branch of that library.

Loading resources. Use libraryResource for scripts and config rather than embedding long strings in Groovy. Be aware that it returns file contents as a string; you write it to the workspace yourself if a tool needs a file path.

Non-serializable work. When a helper must use something CPS cannot handle, such as certain iterators or regex matcher objects, mark the method @NonCPS. Such a method cannot call pipeline steps, so keep it to pure computation. For related thinking on keeping automation modular, see other write-ups on kuryzhev.cloud.

Performance notes

Every build that uses a library performs a checkout of it at the start. For a small repository this is cheap, but it adds up across a busy controller. The library configuration offers an option to cache fetched versions on the controller. That option comes with a refresh interval and a list of versions to exclude from caching. Note that caching interacts badly with floating branch names, because a cached branch can lag behind its latest commits until the cache refreshes. That is another argument for pinning tags. Read the current plugin docs for exact behavior.

The larger cost is usually Groovy execution on the controller. Rules that follow from the CPS model:

Push parsing, filtering, and bulk data work into sh or bat steps on the agent (for example, jq against a JSON file), instead of looping in Groovy.
Avoid very large maps or lists as step return values; they are serialized with the build state.
Prefer one sh block running a short script over dozens of tiny sh calls, since each step adds bookkeeping to the build record.
Load only the libraries a pipeline needs. Each extra @Library adds a checkout and classpath surface.

Pipeline durability settings also trade crash-resilience for write volume. The documented performance-optimized mode reduces disk writes but can lose in-flight state if the controller dies. Decide per job class whether that is acceptable, and verify the options in the Pipeline plugin documentation for your version.

A final practical check: if a library change cannot be explained as "one step, one responsibility, one version tag", the structure has probably started drifting back toward the copy-paste problem it was meant to solve.

DEV Community