Jenkins Configuration as Code with Docker and Kubernetes

So there are a few tools I use in my CI/CD tool-chain - Ansible, GitLab, and Jenkins. They all have their strengths, and weaknesses, it's all about using the right tool for the right job. Sure, you could use a flat-head driver on a Philips but, should you?

In setting these things up time and time, and time, and time, and...

...and time again, I've gotten good at doing it fast, and with as little manual intervention as possible. Most of these tools have some sort of API or CLI to interface with and automate but Jenkins provides a (now more mature) Configuration as Code (JCasC, CaC, etc) plugin. The JCasC plugin allows you to take configuration and store it as YAML, drop it into a directory (or mount into a ConfigMap/Volume in Kubernetes/OpenShift), and Jenkins will pick up that configuration and apply it, so long as the needed plugins are there.

Jenkins Plugins

So, plugins in Jenkins get a bad rep - as do WordPress plugins, and many other libraries, packages, and plugins created by Free Open-Source Software (FOSS) contributors. Sometimes a a plugin doesn't get updated, or a dependency has a vulnerability that is made worse by the parent plugin, or the developer moves on and it just stagnates for years. This is unfortunately the double-edged sword of FOSS, which is why it's often wise to go for supported Open-Source such as that from Red Hat and CloudBees. In fact, I'll be using CloudBees Jenkins Distribution which is a much more properly built Jenkins - fo free.

Anywho, I won't go into the deployment of Jenkins, there are a dozen different ways to do it and to do it properly is a rather large article in and of itself - if there's interest I'll write one though.

Let's assume that you as well have Jenkins, or CloudBees Jenkins Distribution, installed and accessible - great! After the Setup Wizard, one of the first things you'll want to do is start stuffing it with plugins - thankfully I have a handy script for you to do it automatically!

Grab that, drop it into your $JENKINS_HOME (or somewhere wx'able), and pass it a list of plugins you'd like to install, such as:

/var/jenkins_home/jenkins-plugin-stuffer.sh anchore-container-scanner ansible configuration-as-code docker-commons docker-java-api docker-plugin docker-workflow durable-task generic-webhook-trigger git git-client gitlab-plugin git-server kubernetes kubernetes-cli kubernetes-client-api kubernetes-credentials matrix-auth nexus-jenkins-plugin pipeline-aggregator-view pipeline-build-step pipeline-graph-analysis pipeline-input-step pipeline-milestone-step pipeline-model-api pipeline-model-declarative-agent pipeline-model-definition pipeline-model-extensions pipeline-rest-api pipeline-stage-step pipeline-stage-view pipeline-utility-steps role-strategy sonar workflow-aggregator workflow-api workflow-basic-steps workflow-cps workflow-cps-global-lib workflow-durable-task-step workflow-job workflow-multibranch workflow-scm-step

Whew - yeah. Anywho, that's a decent "cloud-native" plugin setup, run that and restart Jenkins, then on to the Configuration as Code ...configuration!

Set JCasC Path

Before the Configuration as Code plugin can load the configuration manifests, you need to tell Jenkins where to load it from! To do so, navigate to the Manage Jenkins > Configuration as Code section of the Web UI.

Add in a directory with where we're going to store the JCasC manifests - I like to do $JENKINS_HOME/jcasc

Click Apply, Save, and now we can start by adding our JCasC manifests!

JCasC for Docker

One of the things you may want to do with your Jenkins instance is build/interact with/run container images via Docker. We've got the plugins set up, now we just need to add the configuration manifest, say a file like $JENKINS_HOME/jcasc/cloud-docker.yml:

jenkins:
  clouds:
    - docker:
        name: "docker"
        dockerApi:
          dockerHost:
            uri: "unix:///var/run/docker.sock"
        templates:
          - labelString: "docker-agent"
            dockerTemplateBase:
              image: "jenkins/slave"
              volumes:
                - /var/run/docker.sock:/var/run/docker.sock
            remoteFs: "/home/jenkins/agent"
            connector:
              attach:
                user: "jenkins"
            instanceCapStr: "10"
            retentionStrategy:
              idleMinutes: 1

This is a great starter configuration if you're using the local Docker Unix Socket (default config when installing Docker on Linux/Mac). Save this file and when Jenkins is restarted the configuration will be loaded allowing you to run Pipelines with the Docker agents. Read more about that here: https://www.jenkins.io/doc/book/pipeline/docker/

JCasC for Kubernetes

Maybe instead of using Docker to build applications and run pipelines in Jenkins, you'd like to use Kubernetes and maybe even Kaniko? No problem, we can do this pretty easily now:

jenkins:
  clouds:
  - kubernetes:
      connectTimeout: 5
      containerCapStr: "10"
      credentialsId: "my-k8s-token"
      serverUrl: "https://k8s.api.example.com"
      skipTlsVerify: true
      jenkinsUrl: "https://jenkins.example.com/jenkins"
      maxRequestsPerHostStr: "32"
      name: "kubernetes"
      readTimeout: 15
      templates:
      - containers:
        - args: "^${computer.jnlpmac} ^${computer.name}"
          image: "registry.access.redhat.com/openshift3/jenkins-slave-maven-rhel7"
          livenessProbe:
            failureThreshold: 0
            initialDelaySeconds: 0
            periodSeconds: 0
            successThreshold: 0
            timeoutSeconds: 0
          name: "jnlp"
          workingDir: "/tmp"
        hostNetwork: false
        label: "maven"
        name: "maven"
        workspaceVolume:
          emptyDirWorkspaceVolume:
            memory: false
        yamlMergeStrategy: "override"
      - containers:
        - args: "^${computer.jnlpmac} ^${computer.name}"
          image: "registry.access.redhat.com/openshift3/jenkins-agent-nodejs-8-rhel7"
          livenessProbe:
            failureThreshold: 0
            initialDelaySeconds: 0
            periodSeconds: 0
            successThreshold: 0
            timeoutSeconds: 0
          name: "jnlp"
          workingDir: "/tmp"
        hostNetwork: false
        label: "nodejs"
        name: "nodejs"
        workspaceVolume:
          emptyDirWorkspaceVolume:
            memory: false
        yamlMergeStrategy: "override"

Save that file as $JENKINS_HOME/jcasc/cloud-kubernetes.yml, restart Jenkins, and enjoy the new ability to deploy pipelines on Kubernetes! Read more about the Jenkinsfile side of things here: https://plugins.jenkins.io/kubernetes/

Wrap

That's that! Within a short time you can deploy Jenkins or CloudBees Jenkins Distribution, add all your plugins with a few commands, set up configuration from code and get to building applications with Jenkins, Docker, and Kubernetes!