DevOps Monitoring and Automation Tool using Jenkins, Prometheus, Grafana and Docker

#devops #jenkins #docker #grafana

In this guide, we will walk through the process of setting up a DevOps monitoring and automation tool and understanding how different tools can be used together for this purpose: Jenkins for continuous integration and continuous deployment(CI/CD), Prometheus for metrics collection, Grafana for visualization, and Docker for containerization.

...

Install Docker

If you haven't already installed Docker, follow the installation instructions for your operating system from the Docker website.

Set up Jenkins

Run Jenkins in a Docker Container:

 ```
 docker run -u 0 --privileged --name jenkins -it -d -p 8080:8080 -p 
 50000:50000 
 \-v/var/run/docker.sock:/var/run/docker.sock 
 \-v $(which docker):/usr/bin/docker 
 \-v /home/jenkins_home:/var/jenkins_home 
 \jenkins/jenkins:latest
```

Access and Setup Jenkins:

Open your browser and go to http://localhost:8080/
You will be requested to provide a password and there is a guide
on where to find it however, in case you can not locate the file
you can run the command: docker logs <docker container id> where
the 'docker container id' is the id of the docker container running
Jenkins, the password will be displayed in the logs.
For this tutorial, you can select to install suggested plugins.
We will install any other required plugins later on.
From the dashboard add a new item, I will be using, Pipeline.

Install Plugins:

Go to "Manage Jenkins" > "Manage Plugins".
In the "Available" tab, search for "Docker Plugin", "Docker Pipeline
Plugin", "Prometheus metrics Plugin" and install them. Restart Jenkins
if prompted.

Set Up Prometheus and Grafana

Create a Docker Network:
docker network create monitoring
Run Prometheus:
Create a prometheus.yml configuration file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'jenkins'
    static_configs:
      - targets: ['jenkins:8080']
      metrics_path: '/prometheus'

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

Run Prometheus with this configuration:

docker run -d --name=prometheus 
\--network=monitoring 
\-p 9090:9090 
\-v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \prom/prometheus

Run Node Exporter:

docker run -d --name=node-exporter 
\--network=monitoring 
\--restart=unless-stopped 
\prom/node-exporter

Run Grafana:

docker run -d --name=grafana 
\--network=monitoring 
\-p 3000:3000 
\grafana/grafana

Configure Grafana

Access Grafana
Open your browser and go to http:localhost:3000.Log in with the default username and password(admin).
Add Prometheus as a Data Source:
Go to "Data Sources" on the left sidebar, "Add data source" and select "Prometheus".
Enter http://prometheus:9090 as the Prometheus server URL.
Click "Save & Test"
Create a Dashboard:
Select "Dashboards" from the left sidebar to create a new dashboard. You can import dashboard using ID or URL. For example ID: '11159'
You can edit one of the panels or 'add visualization', then add a query for the data you want to see.
For this example, we will be looking at how long it takes Jenkins to process a job. Add the query:
default_jenkins_builds_last_build_duration_milliseconds

Automate application deployment with Jenkins

Set Up a Jenkins Pipeline: From the application you had set up in Jenkins, go to "configure" and scroll down to "Pipeline Script" The script below will guide you, you may customize to fit the application you are deploying:

pipeline {
    agent any

    environment {
        DB_HOST = 'your-db-host'
        DB_USER = 'your-db-user'
        DB_PASS = 'your-db-password'
        DB_NAME = 'your-db-name'
    }

    stages {
        stage('Checkout') {
            steps {
                script {
                    checkout([$class: 'GitSCM', branches: [[name: '*/main']],
                      doGenerateSubmoduleConfigurations: false,
                      extensions: [[$class: 'CleanCheckout']],
                      submoduleCfg: [],
                      userRemoteConfigs: [[url: 'https://github.com/yourusername/your-nodejs-typescript-api-repo.git',
                      credentialsId: 'git-credentials-id']]
                    ])
                }
            }
        }

        stage('Install Dependencies') {
            steps {
                sh 'npm install'
            }
        }

        stage('Build') {
            steps {
                sh 'npm run build'
            }
        }

        stage('Test') {
            steps {
                sh 'npm test'
            }
        }

        stage('Build Docker Image') {
            steps {
                script {
                    sh 'docker build -t your-dockerhub-username/your-nodejs-typescript-api:latest .'
                }
            }
        }

        stage('Push Docker Image') {
            steps {
                script {
                    withCredentials([usernamePassword(credentialsId: 'dockerhub-credentials', usernameVariable: 'DOCKER_USERNAME', passwordVariable: 'DOCKER_PASSWORD')]) {
                        sh 'echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin'
                    }
                    sh 'docker push your-dockerhub-username/your-nodejs-typescript-api:latest'
                }
            }
        }

        stage('Deploy') {
            steps {
                script {
                    sh '''
                    docker run -d -p 3000:3000 \
                        -e NODE_ENV=$NODE_ENV \
                        -e DB_HOST=$DB_HOST \
                        -e DB_USER=$DB_USER \
                        -e DB_PASS=$DB_PASS \
                        -e DB_NAME=$DB_NAME \
                        your-dockerhub-username/your-nodejs-typescript-api:latest
                    '''
                }
            }
        }

        stage('Monitor') {
            steps {
                script {
                    // Additional monitoring setup can be added here
                }
            }
        }
    }

    post {
        success {
            echo 'Build succeeded!'
        }
        failure {
            echo 'Build failed!'
        }
    }
}

Credentials

If you will be pulling code from Github you will need to add credentials for git hub and also Docker.
Go to "Manage Jenkins" > "Credentials" select global, then add your credentials for both Github and Docker. Take note of the id as this is what is used as reference in the Pipeline Script.