Jenkins Agent Hardening: Lock Down Build Execution

Originally published on kuryzhev.cloud

---

The scenario

A pull request from a contributor changes a Jenkinsfile, and the build runs on a long-lived agent that also builds your release branch. Nothing in the pipeline looks malicious. However, the agent has a Docker socket mounted, a cloud credentials file in the home directory, and a workspace left over from the last job. Jenkins agent hardening exists to make that situation boring. A build can run arbitrary code, so the machine running it should have nothing worth stealing and no way to reach beyond its own job.

The Jenkins project documents the controller/agent trust model in its controller isolation guide. The short version is that anyone who can change a pipeline can execute code on an agent, so agents are a trust boundary. This guide applies that to four concrete areas:

• keeping builds off the controller
• shrinking agent privileges
• confining permanent agents
• scoping secrets

The examples assume Kubernetes-based ephemeral agents plus one class of permanent Linux agents. Adjust them to your environment, and check the Jenkins and plugin documentation for version-specific behavior.

Prerequisites

• A current Jenkins LTS controller reachable over HTTPS. Agent-to-controller access rules have been always-on since Jenkins 2.326, so there is no legacy toggle to disable.
• Administrator access, ideally with Configuration as Code (JCasC) so the settings below are reviewable in Git.
• The Kubernetes plugin and a cluster where you can create pods in a dedicated namespace (for the ephemeral agents).
• A Linux host with systemd and Java installed (for the permanent agents).
• A test pipeline you can run safely. Do not validate hardening against your production release job.

Also decide up front which jobs are trusted. Pull-request builds from forks and release builds should never share an agent label. Hardening one agent type well is of little use if a single agent type is shared by everything.

Step 1: Keep builds off the controller and shrink the connection surface

The built-in node runs with the controller's filesystem access, including secrets and job configuration. Set its executors to zero so no build can land there.

Then disable the fixed inbound TCP agent port and use WebSocket connections, which travel over your existing HTTPS endpoint.

# jenkins.yaml (Configuration as Code)
jenkins:
  numExecutors: 0        # built-in node runs no builds
  slaveAgentPort: -1     # disables the separate inbound TCP agent port
  mode: EXCLUSIVE        # only jobs with a matching label run on the built-in node

Inbound agents then connect with the -webSocket option. The @ prefix on -secret reads the secret from a file. That keeps it off the command line, where it would show up in the process list. Confirm the exact flags for your agent version with java -jar agent.jar -help.

# Run as the unprivileged agent user, not root
java -jar agent.jar \
  -url https://jenkins.example.com/ \
  -name build-linux-01 \
  -secret @/etc/jenkins-agent/secret \
  -workDir /var/lib/jenkins-agent \
  -webSocket

Watch out for: turning off the TCP port while agents are still configured to use it. Existing inbound agents will fail to reconnect. Migrate them to WebSocket first, or schedule the change in a maintenance window. This includes Kubernetes plugin pod agents: enable the WebSocket option on the Kubernetes cloud configuration before you disable the port.

Step 2: Make agents ephemeral and unprivileged

A fresh pod per build removes the biggest persistence risk: leftover workspaces, caches, and tampered tools. The pod spec should:

• run as a non-root user
• drop Linux capabilities
• avoid mounting the host's container socket

The Kubernetes documentation on security contexts covers each field.

apiVersion: v1
kind: Pod
spec:
  automountServiceAccountToken: false   # builds rarely need Kubernetes API access
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: jnlp                         # default agent container name in the Kubernetes plugin
      image: jenkins/inbound-agent:latest-jdk21   # pin a version or digest in real use
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
      volumeMounts:
        - name: work
          mountPath: /home/jenkins/agent
        - name: tmp
          mountPath: /tmp
  volumes:
    - name: work
      emptyDir: {}
    - name: tmp
      emptyDir: {}

Watch out for: readOnlyRootFilesystem breaks tools that write to the home directory or a cache path. Add an emptyDir for each path that needs writes rather than relaxing the setting.

If builds need to produce container images, use a rootless builder instead of a privileged Docker-in-Docker sidecar. BuildKit in rootless mode and Buildah both work. Avoid Kaniko for new setups, because its original upstream repository has been archived.

A pipeline can define its own pod template. A hardened default therefore does not stop a malicious Jenkinsfile from requesting a privileged pod. Enforce the limits at the cluster level by labeling the agent namespace with pod-security.kubernetes.io/enforce: restricted. Pod Security Admission then rejects pods that violate the Restricted standard. The spec above complies with it.

Pair the pod spec with a Kubernetes NetworkPolicy. Limit egress to the controller, DNS, your artifact registry, and the package mirrors you actually use. NetworkPolicy is only enforced if your CNI plugin supports it. Default-open egress gives a compromised build an easy path to exfiltrate data.

Step 3: Confine permanent agents with systemd

Some workloads, such as hardware tests or macOS and Windows builds, cannot be ephemeral pods. For Linux agents of this kind, run the agent as a dedicated user under a systemd unit that restricts what the process can touch. The directives below are documented in systemd.exec(5). Verify that your distribution's systemd version supports each one. Note that systemd only treats lines beginning with # as comments, so comments cannot follow a value.

# /etc/systemd/system/jenkins-agent.service
[Unit]
Description=Jenkins inbound agent
Wants=network-online.target
After=network-online.target

[Service]
User=jenkins-agent
Group=jenkins-agent
ExecStart=/usr/bin/java -jar /opt/jenkins-agent/agent.jar \
  -url https://jenkins.example.com/ -name build-linux-01 \
  -secret @/etc/jenkins-agent/secret -workDir /var/lib/jenkins-agent -webSocket
# Block privilege escalation via setuid binaries
NoNewPrivileges=true
# Filesystem read-only except ReadWritePaths
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/jenkins-agent
# Empty set: no capabilities
CapabilityBoundingSet=
Restart=on-failure

[Install]
WantedBy=multi-user.target

Make the secret file readable only by the agent user (chmod 600, owned by jenkins-agent). Also clean the workspace between builds, because a permanent agent keeps whatever the last job left behind. Use the cleanWs() step from the Workspace Cleanup plugin, or a scheduled wipe.

Step 4: Scope credentials and label trust levels

Even a hardened agent is only as safe as the secrets it receives. Store credentials in the Jenkins credentials store. Scope them to the folder that needs them rather than globally, and bind them only around the steps that use them. Use separate labels so untrusted builds never share an agent with deploy jobs.

pipeline {
  agent { label 'untrusted-ephemeral' }   // PR builds: no deploy credentials exist here
  stages {
    stage('Test') {
      steps { sh './run-tests.sh' }
    }
    stage('Publish') {
      when {
        beforeAgent true                  // evaluate before allocating the trusted agent
        branch 'main'
      }
      agent { label 'trusted-release' }   // separate pool for release work
      steps {
        withCredentials([string(credentialsId: 'registry-token', variable: 'TOKEN')]) {
          sh 'echo "$TOKEN" | docker login registry.example.com -u ci --password-stdin'
        }
      }
    }
  }
}

Labels in a Jenkinsfile are not a control on their own, because a pull request can edit them. Enforce the separation on the controller:

• Configure your branch source so fork pull requests run the Jenkinsfile from the target branch. The GitHub Branch Source plugin does this through its fork trust setting.
• Restrict which jobs may use the trusted pool, for example with the Job Restrictions plugin or a cloud only the release folder can use.

Watch out for: credential masking only hides exact values in the console log. It does not stop a malicious step from transforming a secret and sending it elsewhere. That is why the trusted pool must be unreachable from untrusted pull requests. Prefer short-lived tokens over static keys, such as OIDC federation to your cloud provider where supported.

Verify and test Jenkins agent hardening

Run checks as the build would, using a throwaway pipeline. Each test should fail in the way you expect. A pass on a forbidden action means the control is missing.

# 1. Identity and privileges inside an ephemeral agent
kubectl -n jenkins-agents exec POD_NAME -c jnlp -- id                       # expect uid 1000, not 0
kubectl -n jenkins-agents exec POD_NAME -c jnlp -- touch /home/jenkins/probe  # expect "Read-only file system"

# 2. Container socket and API access
kubectl -n jenkins-agents exec POD_NAME -c jnlp -- ls /var/run/docker.sock   # expect not found
kubectl -n jenkins-agents exec POD_NAME -c jnlp -- ls /var/run/secrets/kubernetes.io/serviceaccount   # expect not found

# 3. Controller TCP agent port closed (run from a network peer)
nc -zv jenkins.example.com 50000    # use your previous port; expect failure

# 4. Permanent agent sandbox rating
systemd-analyze security jenkins-agent.service

Then complete these checks:

• In the Jenkins UI, confirm the built-in node shows zero executors and that a job with no matching label does not start on it.
• Run a pull-request build that attempts to read a deploy credential ID, and confirm it cannot resolve.
• Try to apply a privileged pod in the agent namespace, and confirm Pod Security Admission rejects it.
• Check that outbound requests to a host outside your NetworkPolicy allowlist time out.

Closing

Jenkins agent hardening is mostly about removing four assumptions:

• that the controller can build
• that agents live forever
• that containers may be root
• that every job deserves every secret

Start with executors set to zero and ephemeral unprivileged pods, which are a practical first step for most setups. Then add systemd confinement and credential scoping for the agents that must persist. Keep the configuration in JCasC and Git so changes are reviewed. Revisit the Jenkins security documentation when you upgrade, since defaults and plugin behavior change between releases. For more infrastructure guides, browse kuryzhev.cloud.

Related

• Cloudflare Origin Hardening: Authenticated Pulls vs IP Allowlist
• Jenkins Shared Library Structure for Reusable Pipelines
• Wordfence Scan Settings: Plugin Scanning vs Server Hardening

Comments

[Lycore Development]

The scenario of a pull request changing a Jenkinsfile on an agent that also builds releases is the one that justifies everything else here. Keeping fork builds and release builds on separate agent labels seems like the highest-value step, with ephemeral pods guaranteeing a clean workspace each time. How do you scope credentials for the release agents: per folder, or injected only into the release job?