Originally published on kuryzhev.cloud
Context: agents are the attack surface, not the controller
We had to harden Jenkins agents after a routine security review turned into something less routine. A pull request from an external contributor triggered a build that, for a brief moment, had a live network path to an internal metadata endpoint it had no business touching. Nothing was exfiltrated. But it was close enough that we stopped what we were doing and audited everything.
For years our mental model was simple: the Jenkins controller has RBAC, matrix-based security, folder permissions — so the controller is the thing we lock down. Agents were "just workers." They ran the build, printed some logs, and got recycled. We assumed that whatever restrictions we set on the controller UI somehow extended down to what code running on an agent could actually do at the OS level.
That assumption is wrong, and it's a common one. Agents execute arbitrary code — your own pipelines, sure, but also third-party fork-PR builds, community plugins pulled during a build step, and whatever a compromised dependency decides to do at build time. If you don't treat that execution environment as hostile by default, you're one bad Jenkinsfile away from a real incident.
What follows is what our audit found, not a theoretical checklist we wrote from a blog post. Three mistakes, each embarrassing in hindsight, and what we changed after.
Mistake 1: We treated agents as trusted extensions of the controller
Our agents connected over standard JNLP/Remoting, and we never restricted what Groovy code running on an agent could call back into the controller JVM. In practice this meant a compromised or malicious build step had a plausible path toward controller-side objects — credentials store included — depending on which plugins were loaded and what libraries a pipeline pulled in.
The bigger problem was label reuse. We had a single pool of agents labeled linux-docker that served both our internal, trusted repositories and external fork-PR builds. Same credentials binding scope, same agent images, same everything. A label in Jenkins is just a scheduling hint — it is not an isolation boundary unless you build one underneath it.
So untrusted code from a stranger's pull request was scheduled onto the same agent pool that had access to deploy credentials for internal services. It never got exploited that we know of, but the exposure was there the entire time, and nobody had explicitly decided to accept that risk. It was just an accident of convenience — one pool was easier to maintain than two.
The fix direction we landed on (details below) was splitting trust tiers before touching anything else: internal builds and PR builds needed to stop sharing infrastructure entirely, not just share it with tighter permissions.
Mistake 2: We mounted docker.sock for build convenience
This one stings the most because we knew better and did it anyway. Several build jobs needed to build and push Docker images, and the fastest way to get Docker-in-Docker working was mounting /var/run/docker.sock into the agent container. It worked immediately. It also meant any build running on that agent had a straightforward path to root on the host node — mounting the socket is functionally equivalent to giving the container root access to the underlying machine.
We didn't get hit by this through an incident. A security review flagged it, and once we understood the blast radius, it was hard to unsee. A single compromised build dependency — a malicious npm package, a poisoned base image, anything running arbitrary code during the build — could pivot from "container" to "node" with almost no extra effort.
Making it worse: our agent containers ran as root by default, with no dropped capabilities and no seccomp or AppArmor profile applied. There was nothing standing between a build process and full control of the container, and from there, the host.
Watch out for this specifically: it's incredibly common in Jenkins-on-Kubernetes setups because docker.sock mounts are the first result in every "how to build Docker images in Jenkins" tutorial. Treat any docker.sock mount on a build node as a critical finding, not a shortcut, per the guidance in Docker's own security documentation.
Mistake 3: We kept long-lived "pet" agents around
Our agent fleet was a mix of static VMs that had been running for months, sometimes years. They were provisioned once from a base image and then patched in place — or, more honestly, patched whenever someone remembered to. Over time they drifted hard from whatever the "declared" image was supposed to look like: manually installed CLI tools, leftover SDK versions, cached credentials from debugging sessions nobody cleaned up.
Workspace hygiene was inconsistent too. Some older pipelines never called cleanWs() at the end of a job, assuming the next build would just overwrite whatever was there. It mostly did — except when it didn't. We found at least one case where a previous job's temporary credentials file survived in the workspace and showed up, unmasked, in a completely unrelated job's build artifacts. Masked console output doesn't help if the secret was written to disk instead of printed to the log.
Patching cadence for these static agents lagged for months at a time because ownership was fuzzy. Nobody's job description said "rebuild the Jenkins agent AMI," so it just didn't happen until a review forced the question. That's the real cost of pet infrastructure — not that it's insecure on day one, but that it quietly gets more insecure every week nobody touches it.
What we do differently now
The single biggest architectural change was moving to ephemeral, pod-per-build agents using the Kubernetes plugin. Every build gets a fresh pod, and the pod is destroyed the moment the build finishes. No leftover workspace, no drifted image, no cached credentials from three jobs ago.
Here's roughly what a hardened pod template looks like for our fork-PR pool today:
# Hardened Kubernetes pod template for a Jenkins build agent
apiVersion: v1
kind: Pod
metadata:
labels:
jenkins/agent-pool: "pr-untrusted" # separate pool from internal builds
spec:
automountServiceAccountToken: false # no implicit cluster API access
securityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
seccompProfile:
type: RuntimeDefault
containers:
- name: build
image: registry.internal/jenkins-agent:2026-week12 # rebuilt weekly
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
limits:
cpu: "2"
memory: "4Gi"
volumeMounts:
- name: workspace
mountPath: /home/jenkins/agent
# deliberately no docker.sock, no hostPath volumes
volumes:
- name: workspace
emptyDir: {} # wiped automatically when pod is destroyed post-build
restartPolicy: Never
Internal and fork-PR builds now run in entirely separate agent pools, enforced through Kubernetes namespaces with a restricted Pod Security Standard profile — no privileged containers, no hostPath, no hostNetwork, full stop. Fork-PR builds get zero long-lived cloud credentials by default; anything they need is a short-lived OIDC or Vault token scoped tightly to that specific job and expired within minutes.
We also cut off unrestricted internet access from build agents with a NetworkPolicy that only allows egress to our package registries and internal artifact store. If a compromised dependency tries to phone home during a build, it has nowhere to go. Agent images are rebuilt on a fixed weekly schedule instead of patched in place, and the Script Console is locked down to a two-person admin group — agents should never need controller-side script execution rights in the first place.
Yes, ephemeral pods add cold-start latency — image pulls and pod scheduling aren't free, and some teams notice the extra 20-40 seconds per build. For anything that touches secrets or third-party code, I think that tradeoff is an easy call. We wrote up a broader take on similar rollout tradeoffs in our DevOps notes on kuryzhev.cloud if you want more context on how we sequenced this migration without blocking releases.
Here's the checklist we now run through before onboarding any new Jenkins agent pool:
Agent hardening checklist (post-incident version):
[ ] No docker.sock or hostPath mounts on any build pod
[ ] Separate agent pools for internal vs. fork-PR / external builds
[ ] Fork-PR builds get zero long-lived cloud credentials
[ ] runAsNonRoot + dropped capabilities on every pod template
[ ] cleanWs() (or pod destruction) guaranteed after every build
[ ] Egress NetworkPolicy limits agents to registries + artifact store
[ ] Agent images rebuilt on a fixed schedule, not patched in place
[ ] Script Console / Groovy approval restricted to a small admin group
[ ] Secrets injected as short-lived tokens (Vault/OIDC), not static creds
[ ] Quarterly review of which labels/pools can reach which credentials
None of this is exotic. It's the same trust-boundary thinking you'd apply to any multi-tenant compute — we just hadn't applied it to Jenkins agents because we'd mentally filed them under "internal tooling" instead of "runs arbitrary third-party code." If you run fork-PR builds, or share agent pools across teams with different trust levels, it's worth doing this audit before something forces you to.
Top comments (0)