Creating a DevSecOps pipeline with Jenkins — Part 1

Hello everyone. I’m Ata, a computer science graduate and currently interested in cybersecurity. In the fast-paced world of cybersecurity, staying ahead of potential threats is crucial. As a junior cybersecurity engineer with a specific interest in DevSecOps, I am excited to share my journey in creating a robust DevSecOps pipeline by using Jenkins and various tools.

I will share my journey in 3 parts since there is too much material and reading them in a single story may be difficult and take a lot of time. In this part, I will talk about DevSecOps pipelines, Jenkins and its installation, SAST and some details about them.

What are DevSecOps Pipelines?

DevSecOps pipelines are similar to DevOps pipelines. They are like assembly lines for building software, but with a big focus on security. These pipelines automate different tasks, like writing code, testing it, putting it into action, and keeping an eye on it afterward. At each step, they also check for security problems, making sure everything stays safe from the start to the end of the process. This way, everyone involved works together to make sure the software is secure right from the beginning.

Jenkins

Jenkins is an open-source automation server widely used for automating various aspects of the software development process, including building, testing, and deploying software. It provides a platform for continuous integration (CI) and continuous delivery (CD), allowing developers to integrate code changes into a shared repository frequently.

Download and Installation

First of all, let’s setup a Jenkins instance on our machine. Since it has a simple installation and using a container can have a negative impact on performance or storage, I installed Jenkins directly on my virtual machine. I’m using Ubuntu 22.04 on VMware Workstation Pro with 16GB of RAM, 100GB of storage and 8 processor cores. These specs are definitely not a limit or requirement.

For the installation of Jenkins I simply followed the Ubuntu/Debian section under Linux section in this link: https://www.jenkins.io/doc/book/installing/. It takes you through the installation of Jenkins itself and Java since Jenkins requires Java to run. You can also find minimal and recommended specs, installation guide, troubleshooting and many other useful information about Jenkins. It is very easy to follow but let me give you a spoiler:

Supposed that you have Java installed in your machine, simply run this command:

sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \\  
 https://pkg.jenkins.io/debian/jenkins.io-2023.key  
echo deb \[signed-by=/usr/share/keyrings/jenkins-keyring.asc\] \\  
 https://pkg.jenkins.io/debian binary/ | sudo tee \\  
 /etc/apt/sources.list.d/jenkins.list > /dev/null  
sudo apt-get update  
sudo apt-get install jenkins

At the end of the installation, you can see that this command has:

• Setup Jenkins as a daemon launched on start,
• Created a ‘jenkins’ user to run this service,
• Directed console log output to systemd-journald,
• Populated /lib/systemd/system/jenkins.service with configuration parameters for the launch, e.g JENKINS_HOME and most importantly,
• Set Jenkins to listen on port 8080.

Configuration

After the installation is complete, go to the Jenkins on your web browser. If you didn’t do any custom configuration, it is probably in http://localhost:8080. This page will greet you:

From the given path, receive the initial password and access Jenkins. You may need root access to reach the path.

Choose “Install suggested plugins”. We will install required plugins for the steps of our pipeline one by one, from the “Available plugins” section that will be mentioned:

After the plugin installation, you are asked to create an admin user for Jenkins. You can choose “Skip and continue as admin” option but you have to use the initial password to access Jenkins. Therefore, I suggest you to create a user. You can use the same credentials as your machine’s credentials for easiness:

You can directly choose “Save and Finish” option since our Jenkins is set up locally instead of on a cloud system and we will conduct all scans locally:

If you are seeing this page, congrats! You have successfully installed Jenkins to your system:

Creating a Pipeline

By clicking on “New Item”, you will get to this page where Jenkins provides many options. On this page, we will choose “Pipeline” option. In this option, we will create our pipeline from scratch, by coding a Jenkinsfile. “Freestyle project” option provides a better UI and easier way to create a pipeline. However, some steps do not fit easily in Freestyle and once you learn how to create a pipeline from scratch, you will automatically learn how to use other options since this is the base of Jenkins pipelines:

On the opened page, you will see the general options for your project. You can leave them like that for now since our purpose is to learn the basics of a security-based scan and we will test a project that is intentionally vulnerable and isn’t a part of a continuing development process. This is the section which we will create the pipeline from the scratch:

Step-by-step Pipeline Development

Now, let’s have an overview of stages we will use in our security scanning pipeline:

• Checkout: Cloning the project from a Git repository (this stage may vary according to our way to access the project).
• Build: Building the project since some steps require a built project.
• SAST: Static Application Security Testing tests upon the code and built project.
• Dependency Check: Performs a scan on dependencies of a project and reports if there are any vulnerabilities on them.
• SBOM: Software Bill Of Materials is a list of components in a piece of software. SBOMs are useful to determine used technologies in a project and can be used for additional security scans.
• SCA: Software Composition Analysis can be used to automate the identification of vulnerabilities in entire container images, packaged binary files and source code.
• Git Secrets Detection: Can prevent accidental code-commit containing a secret.
• Container Security: If there is a container of the project, you can test everything on the container.
• DAST: Dynamic Application Security Testing approaches security from the outside. It requires applications be fully compiled and operational to identify network, system and OS vulnerabilities.

These are our stages that we will implement in our pipeline. Jenkins pipelines use a file type called _Jenkinsfile_. It’s a different file type that you probably used before but it has a simple syntax. For example, this is a single stage Hello World pipeline script:

pipeline {  
    agent any  

    stages {  
        stage('Hello') {  
            steps {  
                echo 'Hello World'  
            }  
        }  
    }  
}

Jenkins allows you to configure many settings according to your needs. However, these parts are the only required ones for a pipeline. Throughout the development of the pipeline, we will use additional parts and syntax rules for tools we will run in stages. For the additional settings and syntax rules you can visit https://www.jenkins.io/doc/book/pipeline/syntax/.

Tip: To add a setting, feature or section, you can use Pipeline Syntax located under Pipeline section:

In Pipeline Syntax, there are useful information about the syntax and most importantly, 2 tools that will assist you: Snippet Generator and Declarative Directive Generator. These tools help you to generate a Jenkinsfile code according your needs and given parameters.

During my learning process, I ran a build for every step to make sure that the code is correct in terms of syntax and configuration. I will follow the same way throughout the note to both show it to you better and remind myself the process.

Checkout

First of all, let’s pull our project we want to scan from a Git repository. For our example project, I chose **Vulnado — Intentionally Vulnerable Java Application**. Here is its link: https://github.com/ScaleSec/vulnado

Every time we start the pipeline or a trigger such as a push or a PR to the repository starts the pipeline, the project will be pulled from the repository. We will use **Git** to perform the pull. Jenkins plugin for Git is installed by default. However, we still need to have Git installed in our machine. To do this, simply run command and install Git binary:

sudo apt-get install git

After this install, you need to invoke the binary in your Jenkins. For this, you can simply add this command to the pipeline:

git ‘https://github.com/ScaleSec/vulnado.git'

This command uses Git plugin for Jenkins to interact with the Git binary in our machine. We will usually interact with tools and programs via such plugins to have a stable and efficient pipeline.

This is our code at the end of this step:

pipeline {  
    agent any  

    stages {  
        stage('Checkout') {  
            steps {  
                git 'https://github.com/ScaleSec/vulnado.git'  
            }  
        }  
    }  
}

Note that we created a stage under “stages” and in this stage, we added the command under “steps”. We will follow the same approach for each stage and commands we need to run.

This is the pipeline dashboard after our first build:

Build

In DevOps and DevSecOps pipelines, building is a required stage to see if the changes on the code didn’t break the program. If project is successfully built, testing stage will start. If the tests are passed, project is finally deployed with a deployment stage. In this DevSecOps pipeline, we are aiming to perform security scans and find the vulnerabilities. In other words, we are doing the testing part of a traditional pipeline. Therefore, we don’t need to bother with building and deploying, yet.

Almost every tool in a DevSecOps pipeline requires project to be built to perform efficient scans on them. Some of them build it themselves, some of them don’t. Therefore, I usually add a stage to build the project since it can be useful and it is easy to add it to the pipeline.

This project is a Maven project. Therefore, I use a command specific to it to build my project. You can use a different command according to your project to be used in the pipeline:

mvn clean package

Note that an install may be required to build your project. For example, I needed to install Java and Maven on my machine since the project I scan is a Maven project.

This is the simple command to run on my terminal to build my project and that’s what we should do in the pipeline too. To run a shell command, simply put single quotes at the start and end of the command and add “sh” at the beginning of the command. This will be the step to run in Jenkins pipeline: sh ‘mvn clean package’\

This is the final code and result of running the pipeline:

pipeline {  
    agent any  

    stages {  
        stage('Checkout') {  
            steps {  
                git 'https://github.com/ScaleSec/vulnado.git'  
            }  
        }  
        stage('Build') {  
            steps {  
                sh 'mvn clean package'  
            }  
        }  
    }  
}

SAST

Here comes the hard part. Previous steps did not require complex tools and processes. We just pulled and build the project. But from now on, we need to use various tools, install them on our machine and make configurations on both the tools and Jenkins itself.

For the SAST stage, I used SonarQube tool. SonarQube is an open-source platform developed by SonarSource for continuous inspection of code quality to perform automatic reviews with static analysis of code to detect bugs and code smells on more than 30 programming languages. I preferred SonarQube instead of other SAST tools because it has a detailed documentation and plugins about integration with Jenkins and SonarQube works with Java projects pretty well. Of course you can similar multi-language-supported tools such as Semgrep or language-specific tools such as Bandit.

Let’s start with the simplest one. We need to download the plugin on Jenkins for SonarQube to connect SonarQube and Jenkins. On the main dashboard of Jenkins, go to Manage Jenkins > Plugins > Available plugins and type “sonarqube”. SonarQube Scanner plugin should appear on the top:

Mark the box next to the plugin and install the plugin. On the installation page, mark the box “Restart Jenkins when installation is complete and no jobs are running” since you need to restart Jenkins eventually to make the plugin available.

Obviously, we must install an instance of SonarQube. You can use a machine on a cloud provider if you want too but for simplicity, we will install a local instance. You can both use a Docker image or the zip file to run SonarQube. Both ways are simple to perform, compatible with Jenkins and this paper. Here is the link and steps for you to follow for installation: https://docs.sonarsource.com/sonarqube/latest/try-out-sonarqube/. The steps in this page is to run a simple instance of SonarQube on a Docker container. If you are curious about it, you can follow the zip file version. Both of them have the same abilities, I just didn’t want to bore you with specific volume and network settings.

You can understand that installation is successful and ready to be used with Jenkins by trying to access SonarQube by accessing URL http://localhost:9000/ (This is the default URL for SonarQube if you didn’t make any custom configuration). This is the first page that will greet you. Your default username and password is “admin”. It will ask you to change your password to continue.

This is the dashboard of SonarQube. Since we don’t have any projects created, it provides us options to choose a DevOps platform. Usually, GitHub is selected but we won’t follow that path. Because choosing GitHub requires us to have a GitHub App. GitHub Apps are tools that extend GitHub’s functionality like opening issues, comment on pull requests, and manage projects. They can also do things outside of GitHub based on events that happen on GitHub:

Using such app may be useful for organizations or frequently developed projects. However, this story’s scope is performing a security test on a project and creation of a GitHub App and integrating it to both SonarQube and Jenkins is difficult and unnecessary for a testing pipeline. Because of this, I want you to choose “Create a local project”. Then, give a name to your project. A key will be suggested to you but any value is okay.

After this page, you can simply choose “Use the global setting” and create your project. It will ask you to choose an analysis method. Choose “With Jenkins”.

In the next page, choose GitHub for “Select your DevOps platform” and it will provide you some steps. We completed some of them. But we didn’t complete some steps about Jenkins and SonarQube integration that are not mentioned here.

For the steps of Jenkins integration, there is a document in SonarQube’s website. However, in my opinion SonarQube documentations are not enough and even sometimes, wrong. Therefore, I will show you the correct steps.

On your Jenkins dashboard, go to Manage Jenkins > Credentials. In this page, click on System > Global credentials (unrestricted) > Add credentials. In here:

• Kind must be Secret Text
• Scope must be Global
• Secret must be token generated on SonarQube. You can easily generate it at User > My Account > Security in SonarQube. Generate a Global Analysis Token and copy and paste it here.

Now, from the Jenkins dashboard again, go to Manage Jenkins > System > SonarQube servers. In here, enter a name to your installation, server URL (which is http://localhost:9000 by default) and server authentication token. This is the credential we just added. You should be able to see it in the dropdown menu. Choose it and save:

Now, the final part. On SonarQube, you can see a script that you can add to your pipeline. However, it needs some adjustments and a fix to work properly. First of all we don’t need def mvn = tool ‘Default Maven’;\ part because our Maven is on the PATH. Because of this, we should change the shell command to: sh “mvn clean verify sonar:sonar -Dsonar.projectKey=vulnado -Dsonar.projectName=’vulnado’”\ Finally, we must include an installation name in the script. It is not mentioned on SonarQube but it is necessary. Installation name is the name you entered in previous step. For example, my installation name is “sonar-local” and here is the part of the code: 

withSonarQubeEnv(installationName: ‘sonar-local’)

pipeline {  
    agent any  

    stages {  
        stage('Checkout') {  
            steps {  
                git 'https://github.com/ScaleSec/vulnado.git'  
            }  
        }  
        stage('Build') {  
            steps {  
                sh 'mvn clean package'  
            }  
        }  
        stage('SonarQube Analysis') {  
            steps{  
                withSonarQubeEnv(installationName: 'sonar-local') {  
                  sh "mvn clean verify sonar:sonar -Dsonar.projectKey=vulnado -Dsonar.projectName='vulnado'"  
                }  
            }  
        }  
    }  
}

This is the result of the build:

The “SonarQube” button simply redirects you to your SonarQube URL. This is the SonarQube page after SAST stage:

Well, this is the end of Part 1. I will upload Part 2 ASAP. In some parts, I didn’t add images to keep the story short. However, I’d love to hear your comments about the format and content of this story, so I can improve the following parts.

Thanks for reading. I hope it helps!