Jenkins Security: Storing Secrets

Jenkins is by far one of my favorite open source tools. Jenkins is an open source automation that that helps and provides support for building and deploying apps. Jenkins Plugins are what enable it to be customized and extended.

You are new to Jenkins? Quickly try it out by running its docker image, with the command

docker run -p 8080:8080 jenkins/jenkins

CI/CD

Automation is fundamental to Continuous Integration and Countinuous Delivery. Jenkins build can be setup for Continuous Integration tasks such as running integration testing. And Continuous Delivery activities such as building a final jar and pushing it to a repository or building a Docker image and pushing it to a registry. However all such integrations require credentials for authorization. Applications tend have a lot of other secrets such as Certificates for authentication for MASSL, credentials for upstream systems, databases and API tokens.

We cannot talk of Continuous Integration and Continuous Delivery (CI/CD) and not talk about automation. Jenkins is a CI tool and its builds can be configured to run Continuous Integration tasks such software integration testing. Not only that but Continuous Delivery actions such as creating a final jar (java app artifact) and pushing it to a repository or even creating a Docker image and pushing it to a registry are examples of Continuous Delivery activities. All of these integrations however need credentials to be able to work. Certificates for authentication for MASSL credentials for upstream systems, databases, and API tokens are common secrets in applications.

Jenkins Plugins for Secret Management

Below are some plugins that allow basic credential management:

• Credentials Binding Plugin: These allow credentials to be bound to environment variables for use in build steps.
• Credentials Plugin: On the other hand, this plugin allows you to store your credentials in Jenkins.

N.B. These plugins are available in default installation of Jenkins.

Below are secrets which are supported by default:

Using secrets in Freestyle Job

For our test, let’s create a Free style Jenkins Project and use Credentials plugins to store our Dockerhub credentials. As seen below, select Username and password (separated) and enter bindings for the credentials. Basically, these are the environment variables that username and password will be available as:

Once that is done, next click on Add and select the right scope and enter Dockerhub credentials:

Now here is an example of how we use our credentials

Using Credentials in Jenkins Pipeline

Now let us look how we can use our credentials in Jenkins Pipeline. In the following stage of our Jenkinsfile we are will be using artifactory-credentials to login to the artifactory and push our docker images to it. Also note that usernameVariable and passwordVariable are available in the withCredentials block:

    stage("docker_push") {
      withCredentials([usernamePassword(credentialsId: 'artifactory-credentials', 
        passwordVariable: 'ARTIFACTORY_KEY', 
        usernameVariable: 'ARTIFACTORY_USER')]) 
      {
        sh "echo $ARTIFACTORY_KEY | docker login -u $ARTIFACTORY_USER --password-stdin ${REGISTRY_URL}"
        sh "docker tag myapp:latest ${REGISTRY_URL}/myapp:${shortCommit}"
        sh "docker push ${REGISTRY_URL}/myapp:${shortCommit}"
      } 

Retrieving Secrets

Jenkins tries to provide some sense of security by masking the credentials in logs. It does that by looking for an exact match and replaces them with asterisks (*****). But note also that this control can simply be bypassed by a user with edit permission by encoding the credentials in the pipeline and printing them in logs.

Please note that the first echo results in our password being masked since it matches our password string.

Now let us decode the encoded password and see what it gives us:

dev@ubu:~ $ echo YWRtaW4xMjM0Cg== | base64 -d
admin1234

Let us explain this. Actually, Jenkins credentials are stored in Jenkins master encrypted by Jenkins instance id. This file usually exists in your linux server at /var/lib/jenkins/credentials.xml. It is therefore very easy to decrypt credentials saved in Jenkins for Jenkins administrators. Moreover, do note that any user that can SSH to Jenkins can read the encrypted credentials file since it has read permissions for everyone. Let us look at that:

dev@ubu:/var/lib/jenkins$ ls -la credentials.xml 
-rw-r--r-- 1 jenkins jenkins 4650 Mar 26 09:28 credentials.xml

This file has passwords encrypted with Jenkins Id:

<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>
    <scope>GLOBAL</scope>
    <id>artifactory-credentials</id>
    <description></description>
    <username>admin</username>
    <password>{AQAAABAAAAAQom3LN7ei0wdm9cdOlGOa4GxDHzpndn0BUPeI4biARto=}</password>
</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>

Our favorite CI-tool, Jenkins provides a handy utility at /script of the URL that can be used to decrypt passwords with the following script:

encryptedPassword = '{AQAAABAAAAAQom3LN7ei0wdm9cdOlGOa4GxDHzpndn0BUPeI4biARto=}'
passwd = hudson.util.Secret.decrypt(encryptedPassword)
println(passwd)

In Conclusion, it is very clear that secrets entered in Jenkins will not be disclosed. Any user that can use a secret in a build can decode and see it. All and any user that can SSH into Jenkins master server can decrypt all secrets using /script utility if they can login to Jenkins Web UI.

Management Issues with Hardcoding Credentials

As we just saw, the principle of least privilege cannot be effectively employed when just using Jenkins Credentials Plugin. Therefore, there has to be a better solution for auditable credential sharing within team, credential rotation and automatic build provisioning that we can use. For that, I would suggest a number of credential management products such as Hashicorp Vault, Cyberark Vault and AWS Secrets Manager. Some open source products like CredStash for AWS help with credential management.