CI/CD & Jenkins

Stages that an Organization Follows in CI/CD:

Jenkins Folder Structure

/var/lib/jenkins/
│
├── 📦 plugins/              → Installed plugins (.jpi files)
│   ├── git.jpi
│   ├── docker.jpi
│   └── slack.jpi
│
├── 💼 jobs/                 → Job configurations only
│   ├── my-app/
│   │   ├── builds/          → Build history
│   │   │   ├── 1/           → Build #1
│   │   │   │   ├── log      → Console output
│   │   │   │   └── build.xml
│   │   │   ├── 2/           → Build #2
│   │   │   └── 3/           → Build #3
│   │   └── config.xml       → Job configuration
│   │
│   └── another-job/
│       ├── builds/
│       └── config.xml
│
├── 🔨 workspace/            → ALL job workspaces
│   ├── my-app/              → my-app workspace
│   │   ├── src/
│   │   ├── package.json
│   │   └── node_modules/
│   │
│   ├── another-job/         → another-job workspace
│   │   ├── src/
│   │   └── pom.xml
│   │
│   └── pipeline-1/          → pipeline-1 workspace
│       └── Dockerfile
│
├── 👥 users/                → User accounts & settings
│   ├── admin/
│   │   └── config.xml
│   └── developer/
│       └── config.xml
│
├── 📝 logs/                 → Jenkins system logs
│
├── 🔐 secrets/              → Encryption keys & secrets
│   ├── master.key
│   └── hudson.util.Secret
│
├── ⚙️ config.xml            → Main Jenkins configuration
│
├── 📊 fingerprints/         → File tracking (artifacts)
│
└── 🗂️ updates/              → Plugin update cache

Jenkins: Plugins vs Tools vs Direct Installation

Aspect	Plugins	Tools (Configured)	Direct Installation
Installed Where	Inside Jenkins	On Jenkins server (Inside Jenkins Directory)	On Jenkins server
Managed By	Jenkins Plugin Manager	Jenkins Tool Config	System admin (apt/yum)
Purpose	Add features/integrations	Version management	Direct execution
UI Integration	✅ Yes	✅ Yes	❌ No
Examples	Git, Slack, Docker Plugin	Maven, JDK, Node.js	Docker, kubectl, AWS CLI
Usage	Special syntax in pipeline	tools {} block	sh 'command'

CI/CD Tools terminology

Concept	Jenkins	GitLab	GitHub	Azure
Top level	Pipeline	Pipeline	Workflow	Pipeline
Phase grouping	Stages	Stages	-	Stages
Execution unit	Stage	Job	Job	Job
Commands	Steps	Script	Steps	Steps
Runs on (where)	Agent	Runner	Runner	Agent/Pool
Keyword	agent	tags	runs-on	pool
Defined at level	Pipeline / Stage	Job	Job (required)	Pipeline / Stage / Job

Installation & Configuration of Jenkins on EC2 Instance

• Go to AWS Console

• Launch instances

**Installation of Jenkins on EC2 Instance

Jenkins is Java based application

Pre-Requisites:

Install Java

sudo apt update
sudo apt install openjdk-11-jre

Verify Java is Installed

java -version

Now, you can proceed with installing Jenkins

curl -fsSL https://pkg.jenkins.io/debian/jenkins.io-2023.key | sudo tee \
  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins

Changing EC2 inbound traffic rules to expose Jenkins

Note: By default, Jenkins will not be accessible to the external world due to the inbound traffic restriction by AWS. Open port 8080 in the inbound traffic rules as show below.

• EC2 > Instances > Click on

• In the bottom tabs -> Click on Security

• Security groups

• Add inbound traffic rules as shown in the image (you can just allow TCP 8080 as well, in my case, I allowed All traffic).

Login to Jenkins using the below URL:

http://[public IP of ec2]:8080

Note: If you are not interested in allowing All Traffic to your EC2 instance

1. Delete the inbound traffic rule for your instance

2. Edit the inbound traffic rule to only allow custom TCP port 8080

After you login to Jenkins,

• Run the command to copy the Jenkins Admin Password - sudo cat /var/lib/jenkins/secrets/initialAdminPassword

• Enter the Administrator password

Click on Install suggested plugins

Wait for the Jenkins to Install suggested plugins

Create First Admin User or Skip the step [If you want to use this Jenkins instance for future use-cases as well, better to create admin user]

Jenkins Installation is Successful. You can now starting using the Jenkins

Install the Docker Pipeline plugin in Jenkins:

• Log in to Jenkins.

• Go to Manage Jenkins > Manage Plugins.

• In the Available tab, search for "Docker Pipeline".

• Select the plugin and click the Install button.

• Restart Jenkins after the plugin is installed.

Wait for the Jenkins to be restarted.

Docker Slave Configuration

Run the below command to Install Docker

sudo apt update
sudo apt install docker.io

Grant Jenkins user and Ubuntu user permission to docker deamon.

sudo su - 
usermod -aG docker jenkins
usermod -aG docker ubuntu
systemctl restart docker

Once you are done with the above steps, it is better to restart Jenkins.

http://<ec2-instance-public-ip>:8080/restart

The docker agent configuration is now successful.

Jenkinsfiles Structure

ENV Variables

<jenkins-url>/env-vars.html

Jenkins Credentials

A secure, centralized store for sensitive data (passwords, tokens, SSH keys, certificates) — so you never hardcode secrets in your pipelines.

Types of Credentials

Type	Use Case
Username & Password	DB logins, basic auth
Secret Text	API tokens, secret strings
SSH Username with Private Key	Git SSH, server access
Secret File	.env files, certs
Certificate	PKCS#12 keystores

Credentials code

• Credentials Binding Plugin (pre-requisite)
• Defined Credentials in Jenkins UI
• credentials("credentialsId") binds the credentials to our env block variables.

environment global block

Declares credentials as environment variables — accessible throughout the entire pipeline

environment {
    // Binds 'credentialsId' credentials to SERVER_CRED
    // Jenkins auto-creates: SERVER_CRED_USR (username) & SERVER_CRED_PSW (password)
    SERVER_CRED = credentials("credentialsId")
}

withCredentials scope block

Temporarily injects credentials into a specific block — variables only available within the block scope

withCredentials([
    usernamePassword(
        credentialsId: 'server-cred',   // ID defined in Jenkins credentials store
        usernameVariable: 'USER',        // Username bound to $USER
        passwordVariable: 'PASS'         // Password bound to $PASS
    )
]) {
    sh "user ${USER}, ${PASS}"           // Credentials used inside shell step
}

Sample Jenkinsfile- Docker as Agent

jenkinsfile:

pipeline {
  agent {
    docker { image 'node:16-alpine' }
  }

  // Define parameters for the pipeline
  parameters {
    choice(name: 'ENV', choices: ['dev', 'test', 'prod'], description: 'Select environment') // choice parameter
    booleanParam(name: 'RUN_TESTS', defaultValue: true, description: 'Run tests?') // boolean parameter
  }

  // Make available maven command in stages
  tools {
    maven 'Maven' 
  }

  // User defined ENV
  environment {
    APP_ENV = "test"
  }

  stages {
    stage('Checkout') {
      steps {
        git url: 'https://github.com/your-repo.git', branch: 'main'
      }
    }

    stage('Parallel Quality Checks') {
      // parallel execution of jobs
      parallel {
        stage('Lint') {
          steps {
            sh 'npm run lint'
          }
        }

        stage('Test') {
          // Run this stage only if ENV is dev and RUN_TESTS is true from parameters
          when {
            expression {
              params.ENV == 'dev' && params.RUN_TESTS
            }
          }
          steps {
            sh 'npm test'
          }
        }

    stage('Deploy') {
      // This stage will pause the pipeline and wait for user input before proceeding with the deployment. (Scope parameter block)
      input {
        message 'Select deployment environment' // The message displayed to the user when prompting for input
        ok 'Done' // The 'ok' button will be labeled as 'Done'
        parameters {
          choice(name: 'ENV', choices: ['Staging', 'Production'], description: 'Select the deployment environment')
        }
      }

      steps {
        echo "Deploying to ${ENV} environment..."
      }
    }

  // Post actions to run after stages
  post {
    success {
      echo 'Success!'
    }
    failure {
      echo 'Failed!'
    }
  }
}

Groovy Scripts

Use Groovy scripts in Jenkinsfile in order to reduce complexity.

Groovy Script

vars/myScript.groovy

def buildApp() {
    echo "Building application..."
}

def test() {
    echo "Testing application..."
}

return this //  Makes the script object itself available for method calls

Jenkinsfile

pipeline {
    agent any

    environment {
        gv = load 'vars/myScript.groovy'
    }

    stages {
        stage('Build App') {
            steps {
                script {
                    gv.buildApp()
                }
            }
        }

        stage('Run Tests') {
            steps {
                script {
                    gv.test()
                }
            }
        }

    }
}

Jenkins Backup

EBS Snapshot (Best for disaster recovery)

EC2
 └── EBS Volume
      └── Snapshot

Snapshot → New Volume → Attach to EC2

S3 Backup (App level)*

/var/lib/jenkins
   ↓
tar.gz
   ↓
S3 Bucket

Jenkins Pipeline for Java based application using Maven, SonarQube, Argo CD, Helm and Kubernetes

Jenkins Pipeline for Java based app

🌟 Enjoy Learning! 😊