DEV Community

David
David

Posted on

Expense Tracker: Learning CI/CD

#cicd #jenkins #springboot #linux

(Originally published on Medium)

Introduction

At some point, I had no idea what CI and CD were about, so I decided to learn about them. I got to learn enough to be able to do a simple Build-Test-Deploy cycle, so now I know enough to define them in my own words:

Continuous Integration: JUnit tests are run on every commit to the dev branch
Continuous Delivery: the JAR is also built and sent straight to the test server when the tests pass, with potential smoke tests such as a “User can log in” test being run.
Continuous Deployment: the build is also sent to production after the smoke tests pass
Sten from Atlassian does a great job of explaining the differences between the 3 terms

Atlassian CI CD article image

My tool of choice is Jenkins, mainly because I prefer a self-hosted approach, and I’ll talk about the Jenkinsfile I wrote for Expense Tracker.

Architecture overview

I have 2 main VM’s, both of which are running Ubuntu

  1. Controller This is where Jenkins itself runs.
  2. Test Server This is where the “dev” profile of the app will run. It’s also where I’ll be running my tests with the “ci” profile.

Jenkinsfile Walk-through

So at a basic level, the tasks mainly involved:

  1. Resetting the database
  2. Generating the relevant code
  3. Testing
  4. Building
  5. Deploying

MySQL Credentials

Using credentials

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

favicon jenkins.io 

environment {
  MYSQL_CREDS = credentials('mysql_credentials')
}
Enter fullscreen mode Exit fullscreen mode

The credentials used here consist of a username and password combination stored within Jenkins. Other credential types I use in the file include Secrets Text for Ansible and a Secrets file for the Spring configuration. I did this to make sure I never have to commit my credentials directly to source control. HashiVault is also an option, but I decided it wasn't needed for my use case.

Use agent “Test Server”

Using Jenkins agents

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

favicon jenkins.io

Managing Nodes

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

favicon jenkins.io 

agent {
  node {
    label 'Test Server'
  }
}
Enter fullscreen mode Exit fullscreen mode

Here, I used ssh-keygen to generate a private key and copy it over as a secret into Jenkins.

Flyway 

stage('Flyway') {
  steps {
    sh 'chmod +x ./gradlew'
    sh './gradlew setUpCIDb'
    dir ('database/flyway') {
      sh 'flyway migrate -environment=ci  -user=$MYSQL_CREDS_USR -password=$MYSQL_CREDS_PSW'
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Here, I drop and recreate the database before running migrations on it. There’s an exclusive database for this task, and it is suffixed with “_ci”. Flyway was downloaded and symlinked on this machine into /usr/bin for this to work, and I created an environment just for this

jOOQ 

stage('JooQ') {
  steps {
    sh './gradlew jooqCodeGenCi'
  }
Enter fullscreen mode Exit fullscreen mode

jOOQ enables me to work with a schema-first approach for the database by generating classes based on the tables and views I have, so this block is necessary for all the needed classes to be present.

System properties governing code generation

System properties governing code generation

favicon jooq.org

The main issue I had here was that even though jOOQ supports using command-line properties, it didn’t work until I used the driver, url, username, and password all on the command line. Mixing command-line authentication with configuration file properties didn’t work.

tasks.register('jooqCodeGenCi', Exec) {
    dependsOn jooqCodeGenCopy

    workingDir "${rootDir}/database/jooq"
    commandLine "java", "-Djooq.codegen.propertyOverride=true", 
      "-Djooq.codegen.jdbc.driver=com.mysql.jdbc.Driver", 
      "-Djooq.codegen.jdbc.url=jdbc:mysql://172.26.48.26:3306/expense_tracker_database_ci", 
      "-Djooq.codegen.jdbc.username=${System.getenv("MYSQL_CREDS_USR")}", 
      "-Djooq.codegen.jdbc.password=${System.getenv("MYSQL_CREDS_PSW")}", 
      "-Djooq.codegen.propertyOverride=true", 
      "-cp", "*",
      "org.jooq.codegen.GenerationTool", 
      "configuration-ci.xml"

}
Enter fullscreen mode Exit fullscreen mode

OpenAPI 

stage('Open API') {
  steps {
    sh './gradlew openApiGenerate'
  }
}
Enter fullscreen mode Exit fullscreen mode

Similar to jOOQ, I use an API-first or spec-first approach in my code, meaning that all of the entities and paths that my controllers rely on need to be generated first.

Documentation for the spring Generator | OpenAPI Generator

METADATA

favicon openapi-generator.tech

https://github.com/OpenAPITools/openapi-generator/blob/master/modules/openapi-generator-gradle-plugin/README.adoc

Build 

stage('Build') {
  steps {
    sh './gradlew assemble'
  }
}
Enter fullscreen mode Exit fullscreen mode

Compiles the classes

Test

Using environment variables

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

favicon jenkins.io

Pipeline Syntax

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

favicon jenkins.io 

environment {
  SPRING_PROFILES_ACTIVE='ci'
}
Enter fullscreen mode Exit fullscreen mode

I start by setting the active Spring Profile to “ci”, one that I’ve made exclusively for this pipeline.

sh 'rm -f app/src/main/resources/secrets-ci.properties'
sh 'chmod o+w -R app'
withCredentials([file(credentialsId: 'expense-tracker-backend-secrets-ci', variable: 'expenseTrackerApplicationSecrets')]) {
  sh 'cp $expenseTrackerApplicationSecrets app/src/main/resources/secrets-ci.properties'
}
sh './gradlew test'
Enter fullscreen mode Exit fullscreen mode

I then remove any potentially existing secrets file before copying over a new one from Jenkins credentials. What I store there includes:

  • The JDBC password
  • The JWT private key

The -f flag is to ensure an exit code of 0 if the file does not exist, because with a non-zero exit code, the pipeline will fail.

And then I test. How exactly I test is discussed in the previous article

Flyway Dev 

stage('Flyway Dev') {
  steps {
    dir ('database/flyway') {
      sh 'flyway migrate -environment=dev  -user=$MYSQL_CREDS_USR -password=$MYSQL_CREDS_PSW'
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Exactly like my first Flyway migration, but this time for my “dev” environment. There’s no DROP-ing this time, though

Since there were no issues with the CI migration, I’m confident there will be no issues with this migration.

Ansible 

stage('Ansible setup') {
  agent {
    node {
      label 'built-in'
    }
  }
  environment {
    ANSIBLE_BECOME_PASS = credentials('ansible-become-password')
  }
  steps {
    dir ('ansible') {
      sh 'ansible-playbook -i inventory playbooks/setup_service.yaml  --private-key ~/.ssh/jenkins_agent_key -e "ansible_become_password=\'$ANSIBLE_BECOME_PASS\'"'
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

I won’t go into the full Ansible playbook here, but it essentially configures my app to run with the OpenTelemetry Java Agent and Pyroscope using SystemD.

I needed to run this off the Jenkins host machine, which is Controller in this case — the built-in node, and my main issue was finding the name to use from the docs. I found it in a forum instead.

Deploy 

stage('Deploy') {
  steps {
    sh './gradlew bootJar'
    sh 'cp "./app/build/libs/expense-tracker-backend.jar" /opt/expense_tracker/expense-tracker-backend.jar'
    sh 'cp app-version /opt/expense_tracker/app-version'
    sh 'chmod 777 /opt/expense_tracker/expense-tracker-backend.jar'
    withCredentials([file(credentialsId: 'expense-tracker-backend-secrets-dev', variable: 'expenseTrackerApplicationSecrets')]) {
      sh 'sudo rm -f /opt/expense_tracker/secrets-dev.properties'
      sh 'cp $expenseTrackerApplicationSecrets /opt/expense_tracker/secrets-dev.properties'
      sh 'chmod 700 /opt/expense_tracker/secrets-dev.properties'
      sh 'sudo chown root:root /opt/expense_tracker/secrets-dev.properties'
    }
    sh 'sudo systemctl enable expense_tracker_backend'
    sh 'sudo systemctl daemon-reload'
    sh 'sudo systemctl restart expense_tracker_backend'
    sh 'sudo systemctl status expense_tracker_backend'
  }
}
Enter fullscreen mode Exit fullscreen mode

The only really interesting parts in this block are here:

sh 'sudo systemctl enable expense_tracker_backend'
sh 'sudo systemctl daemon-reload'
sh 'sudo systemctl restart expense_tracker_backend'
sh 'sudo systemctl status expense_tracker_backend'
Enter fullscreen mode Exit fullscreen mode

Usually, sudo can’t be run without a password prompt, and this is a problem for an automated script. However, I can get it to work for specific commands using the sudoers file. The directive needs to be the last in the file.

jenkins ALL=(ALL) NOPASSWD: /usr/bin/systemctl, /usr/bin/chown, /usr/bin/rm
Enter fullscreen mode Exit fullscreen mode

I made sure to only grant access to the commands I needed and nothing more.

I had cases where the service would restart but still crash due to a mistake I made, like some improper configuration. The status command is there to fail and make sure I know something went wrong.

Post-Script Execution 

post {
  always {
    junit 'app/build/test-results/**/*.xml'
  }
}
Enter fullscreen mode Exit fullscreen mode

Recording tests and artifacts

Jenkins – an open source automation server which enables developers around the world to reliably build, test, and deploy their software

favicon jenkins.io

Here I export the results of the test run from the agent so they can be viewed from within Jenkins.

And here’s the full Jenkinsfile:

pipeline {
    environment {
        MYSQL_CREDS = credentials('mysql_credentials')
    }
    agent {
        node {
            label 'Test Server'
        }
    }
    // Branch dev
    stages {

        stage('Flyway') {

            steps {
                sh 'chmod +x ./gradlew'
                sh './gradlew setUpCIDb'
                dir ('database/flyway') {
                    sh 'flyway migrate -environment=ci  -user=$MYSQL_CREDS_USR -password=$MYSQL_CREDS_PSW'
                }
            }
        }
        stage('JooQ') {
            steps {
                sh './gradlew jooqCodeGenCi'
            }
        }
        stage('Open API') {
            steps {
                sh './gradlew openApiGenerate'
            }
        }
        stage('Build') {
            steps {
                sh './gradlew assemble'
            }
        }
        stage('Test') {
            environment {
                SPRING_PROFILES_ACTIVE='ci'
            }
            steps {
                sh 'rm -f app/src/main/resources/secrets-ci.properties'
                sh 'chmod o+w -R app'
            withCredentials([file(credentialsId: 'expense-tracker-backend-secrets-ci', variable: 'expenseTrackerApplicationSecrets')]) {
                sh 'cp $expenseTrackerApplicationSecrets app/src/main/resources/secrets-ci.properties'
            }
                sh './gradlew test'
            }
        }
        stage('Flyway Dev') {
            steps {
                dir ('database/flyway') {
                    sh 'flyway migrate -environment=dev  -user=$MYSQL_CREDS_USR -password=$MYSQL_CREDS_PSW'
                }
            }
        }
        stage('Ansible setup') {
            agent {
                node {
                    label 'built-in'
                }
            }
            environment {
                ANSIBLE_BECOME_PASS = credentials('ansible-become-password')
            }
            steps {
                dir ('ansible') {
                    sh 'ansible-playbook -i inventory playbooks/setup_service.yaml  --private-key ~/.ssh/jenkins_agent_key -e "ansible_become_password=\'$ANSIBLE_BECOME_PASS\'"'
                }
            }
        }
        stage('Deploy') {
            steps {
                sh './gradlew bootJar'
                sh 'cp "./app/build/libs/expense-tracker-backend.jar" /opt/expense_tracker/expense-tracker-backend.jar'
                sh 'cp app-version /opt/expense_tracker/app-version'
                sh 'chmod 777 /opt/expense_tracker/expense-tracker-backend.jar'
                withCredentials([file(credentialsId: 'expense-tracker-backend-secrets-dev', variable: 'expenseTrackerApplicationSecrets')]) {
                    sh 'sudo rm -f /opt/expense_tracker/secrets-dev.properties'
                    sh 'cp $expenseTrackerApplicationSecrets /opt/expense_tracker/secrets-dev.properties'
                    sh 'chmod 700 /opt/expense_tracker/secrets-dev.properties'
                    sh 'sudo chown root:root /opt/expense_tracker/secrets-dev.properties'
                }
                sh 'sudo systemctl enable expense_tracker_backend'
                sh 'sudo systemctl daemon-reload'
                sh 'sudo systemctl restart expense_tracker_backend'
                sh 'sudo systemctl status expense_tracker_backend'
            }
        }
    }
    post {
            always {
                junit 'app/build/test-results/**/*.xml'
            }
    }
}
Enter fullscreen mode Exit fullscreen mode

And a sample success run

Satisfactory button-click output

Conclusion

There are several areas for improvement:

  • My app version is manually updated right now, and my own forgetfulness means that I’d rather automate incrementing it.
  • All of this was done using the Git daemon. I’d rather set up Gitea so I can use webhooks instead of manually clicking build whenever I commit to dev
  • I could also use an artifact management tool like Artifactory or Sona Nexus to keep the past few JAR files in case I need them

But I’m also enjoying some worthwhile benefits:

  • The build verification phase happens automatically
  • I don’t have to ssh and scp the JAR file manually

Now that the application backend is up and running, next I’ll be talking about how I use OpenTelemetry to ensure I have full visibility into both ends of the app at runtime, starting with Prometheus.

Thank you for your time

Top comments (0)