Setting Up DevOps Monitoring and Automation with Jenkins, Prometheus, Grafana, and Docker

#devops #monitoring #docker #containers

In this guide, we'll explore how to create a DevOps monitoring and automation system using Jenkins for CI/CD, Prometheus for metrics collection, Grafana for data visualization, and Docker for containerization.

Installing Docker
First, ensure Docker is installed on your system. Follow the installation instructions provided on the Docker website for your specific operating system.

Setting Up Jenkins
To run Jenkins in a Docker container, execute the following command:

docker run -u 0 --privileged --name jenkins -it -d -p 8080:8080 -p 50000:50000 \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $(which docker):/usr/bin/docker \
-v /home/jenkins_home:/var/jenkins_home \
jenkins/jenkins:latest

Access and Configure Jenkins
Open your browser and navigate to http://localhost:8080.
Provide the initial admin password, which can be found by running docker logs .
Install the suggested plugins during the initial setup.
From the Jenkins dashboard, create a new item and choose "Pipeline".

Install Required Plugins
Navigate to "Manage Jenkins" > "Manage Plugins" and install the following plugins:

Docker Plugin
Docker Pipeline Plugin
Prometheus Metrics Plugin
Restart Jenkins if prompted.

Setting Up Prometheus and Grafana
Create a Docker Network

docker network create monitoring

Run Prometheus
Create a prometheus.yml configuration file with the following content:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'jenkins'
    static_configs:
      - targets: ['jenkins:8080']
      metrics_path: '/prometheus'

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

Run Prometheus with this configuration:

docker run -d --name=prometheus --network=monitoring -p 9090:9090 \
-v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Run Node Exporter

docker run -d --name=node-exporter --network=monitoring --restart=unless-stopped prom/node-exporter

Run Grafana

docker run -d --name=grafana --network=monitoring -p 3000:3000 grafana/grafana

Configuring Grafana
Access Grafana
Open your browser and go to http://localhost:3000. Log in with the default credentials (username: admin, password: admin).

Add Prometheus as a Data Source
Navigate to "Data Sources" on the left sidebar, click "Add data source," and select "Prometheus".
Enter http://prometheus:9090 as the Prometheus server URL.
Click "Save & Test".
Create a Dashboard
Select "Dashboards" from the left sidebar to create a new dashboard.
Import a dashboard using an ID or URL (e.g., ID: 11159).
Add a query to visualize Jenkins job duration:

default_jenkins_builds_last_build_duration_milliseconds

Automating Application Deployment with Jenkins
Set Up a Jenkins Pipeline
From your Jenkins dashboard, navigate to your pipeline project, go to "Configure," and add the following script under "Pipeline Script":

pipeline {
    agent any

    environment {
        DB_HOST = 'your-db-host'
        DB_USER = 'your-db-user'
        DB_PASS = 'your-db-password'
        DB_NAME = 'your-db-name'
    }

    stages {
        stage('Checkout') {
            steps {
                script {
                    checkout([$class: 'GitSCM', branches: [[name: '*/main']],
                      doGenerateSubmoduleConfigurations: false,
                      extensions: [[$class: 'CleanCheckout']],
                      submoduleCfg: [],
                      userRemoteConfigs: [[url: 'https://github.com/yourusername/your-nodejs-typescript-api-repo.git',
                      credentialsId: 'git-credentials-id']]
                    ])
                }
            }
        }

        stage('Install Dependencies') {
            steps {
                sh 'npm install'
            }
        }

        stage('Build') {
            steps {
                sh 'npm run build'
            }
        }

        stage('Test') {
            steps {
                sh 'npm test'
            }
        }

        stage('Build Docker Image') {
            steps {
                script {
                    sh 'docker build -t your-dockerhub-username/your-nodejs-typescript-api:latest .'
                }
            }
        }

        stage('Push Docker Image') {
            steps {
                script {
                    withCredentials([usernamePassword(credentialsId: 'dockerhub-credentials', usernameVariable: 'DOCKER_USERNAME', passwordVariable: 'DOCKER_PASSWORD')]) {
                        sh 'echo $DOCKER_PASSWORD | docker login -u $DOCKER_USERNAME --password-stdin'
                    }
                    sh 'docker push your-dockerhub-username/your-nodejs-typescript-api:latest'
                }
            }
        }

        stage('Deploy') {
            steps {
                script {
                    sh '''
                    docker run -d -p 3000:3000 \
                        -e NODE_ENV=$NODE_ENV \
                        -e DB_HOST=$DB_HOST \
                        -e DB_USER=$DB_USER \
                        -e DB_PASS=$DB_PASS \
                        -e DB_NAME=$DB_NAME \
                        your-dockerhub-username/your-nodejs-typescript-api:latest
                    '''
                }
            }
        }

        stage('Monitor') {
            steps {
                script {
                    // Additional monitoring setup can be added here
                }
            }
        }
    }

    post {
        success {
            echo 'Build succeeded!'
        }
        failure {
            echo 'Build failed!'
        }
    }
}

Adding Credentials
Add credentials for GitHub and Docker in Jenkins:

Navigate to "Manage Jenkins" > "Credentials".
Select "Global" and add your credentials for GitHub and Docker. Note the ID, as it will be referenced in the pipeline script.
By following these steps, you can set up a comprehensive DevOps monitoring and automation tool using Jenkins, Prometheus, Grafana, and Docker.
Build Application
Build your application and view the metrics on the Grafana dashboard 😃.