Gitlab Integration with Sonarqube: Automate your quality gates

Martin Huter — Fri, 27 Sep 2019 15:00:27 +0000

TL/DR: A new parameter made this post for a lot of use cases obsolete. You can activate sonar.qualitygate.wait to have a built in waiting mechanism which exits the plugin with a non-zero return code on a failing quality gate. More information at: https://docs.sonarqube.org/latest/analysis/gitlab-cicd/

I accidentally killed the first release of this article so here its up again. Any improvement suggestions very welcome.

This article was originally posted on my companies blog at viesure.io/blog.

Last week I took the task of our sprint to create a build breaker step for Sonar in our GitLab deployment pipeline.
To clarify this term: We wanted each build to break if the quality gate of Sonar fails.
Since this was not straightforward, I wanted to share my result.
I am using the API of the SonarQube server to get the information about the status of the quality gate.

Preconditions to use the script

A Sonar API Token that has sufficient rights to access the data of the analysis. If you don't know where to get the token look it up in the documentation
A sonar runner needs to be executed before, to get a ceTaskId for the requests. For maven the sonar:sonar runner creates a file at target/sonar/report-task.txt.

#Example content of the report-task.txt
projectKey=io.viesure:project
serverUrl=https://sonar.server.url
serverVersion=7.7.0.23042
branch=master
dashboardUrl=https://sonar.server.url/dashboard?id=io.viesure%3Aproject&branch=master
ceTaskId=AWy0noy7Aqf9WEmmjKgw
ceTaskUrl=https://sonar.server.url/api/ce/task?id=AWy0noy7Aqf9WEmmjKgw

A detailed explanation of the script

In this section, I will explain the parts of the script step by step.
TL;DR at the end is a link to the full script on GitHub.

Semi-static parameters

Since we are using Sonar with Maven in a CI/CD pipeline, I hard link the source for the result. Depending on your runner, this might differ. A better practice would also be to also extract the sonar server property to the CI environment variables, to reduce the nonfunctional information within the code.

#!/usr/bin/env sh

SONAR_RESULT=${2:-"target/sonar/report-task.txt"}
SONAR_SERVER=${1:-"https://our.sonar.server"}

Sonar API Token

We are checking if the environment has set the sonar API token. If this variable is not available, we terminate the script with a proper error message.

if [ -z $SONAR_API_TOKEN ]
then
  echo "Sonar API Token not set."
  exit 1
fi

Extract ceTaskId from Sonar result and validate access rights

First, we are checking if the sonar result file exists.

if [ ! -f $SONAR_RESULT ]
then
  echo "Sonar result does not exist"
  exit 1
fi

Afterwards, we use sed to extract the task id for further processing. The task id is a unique hash that refers to a sonar runner result that is processed by the server.

CE_TASK_ID=`sed -n 's/ceTaskId=\(.*\)/\1/p' < $SONAR_RESULT`

if [ -z $CE_TASK_ID ]
then
  echo "ceTaskId is not set from sonar build."
  exit 1
fi

Finally, we are checking if our API Token has proper access rights to get information about the processing state of the found task id.

HTTP_STATUS=$(curl -s -o /dev/null -w '%{http_code}' -u $SONAR_API_TOKEN: $SONAR_SERVER/api/ce/task\?id\=$CE_TASK_ID)

if [  "$HTTP_STATUS" -ne 200 ]
then
  echo "Sonar API Token has no access rights."
  exit 1
fi

Find analysisId for a taskId

After processing a project with a sonar runner the SonarQube server processes the result to generate an analysis report. The task.analysisId from the response JSON is available after the process has finished. For this reason, I created a simple time loop to evaluate the analysisId from the response. Jq returns null if the property that is filtered is not part of the JSON.

ANALYSIS_ID=$(curl -XGET -s -u $SONAR_API_TOKEN: $SONAR_SERVER/api/ce/task\?id\=$CE_TASK_ID | jq -r .task.analysisId)
I=1
TIMEOUT=0
while [ $ANALYSIS_ID = "null" ]
do
  if [ "$TIMEOUT" -gt 30 ]
  then
    echo "Timeout of " + $TIMEOUT + " seconds exceeded for getting ANALYSIS_ID"
    exit 1
  fi
  sleep $I
  TIMEOUT=$((TIMEOUT+I))
  I=$((I+1))
  ANALYSIS_ID=$(curl -XGET -s -u $SONAR_API_TOKEN: $SONAR_SERVER/api/ce/task\?id\=$CE_TASK_ID | jq -r .task.analysisId)
done

Get the quality gate information

The quality gate API endpoint returns a JSON that contains all parts of the quality analysis from Sonar. The projectStatus.status is representing the state of the quality gate and was our desired health metric. If you want to have a different setup, adapt the jq processing to your needs and find your breaking parameter.

STATUS=$(curl -XGET -s -u $SONAR_API_TOKEN: $SONAR_SERVER/api/qualitygates/project_status?analysisId=$ANALYSIS_ID | jq -r .projectStatus.status)

if [ $STATUS = "ERROR" ]
then
  echo "Qualitygate failed."
  exit 1
fi

echo "Sonar Qualitygate is OK."
exit 0

Executing the script in the pipeline

First, execute the script after the sonar execution.

#!/usr/bin/env sh

export MAVEN_OPTS="-Dmaven.repo.local=maven.repository -Dsonar.branch.name=$CI_COMMIT_REF_NAME"

./mvnw sonar:sonar
. ${SCRIPT_DIR}/sonar_break_build.sh

Then define a job that executes sonar and the breaker.

sonar:
    stage: sonar
    tags: [java11]
    dependencies:
        - mvn-build
    script:
        - ${SCRIPT_DIR}/mvn_sonarqube.sh
    only:
        - branches
    cache:
        paths:
            - maven.repository/

More detailed information about the script can is available at our blog post about optimizing our build times with gitlab on aws

Conclusion

Our goal is to deliver a good maintainable code as fast as possible.
To achieve this fast feedback is crucial and breaking our build, on the first commit that introduces new security issues, bugs and code smells, is game-changing for this goal.

Some colleagues might be annoyed at first, that their code is instantly analyzed or they have to fix stuff that does not pass the quality gate right away. On the upside code that does not fit the quality standard won't ever be merged to develop or master.
Also, you now need to interact with Sonar more often, but this can also be an advantage to shape your ruleset to the degree that everybody can live with it on a daily basis.
Another question on fine-tuning that came up was if we should break the build on Sonar or mark it as unstable to minimize the disruption of the pipeline, but still get the feedback from Sonar.

I hope you enjoyed the read.
Additional, I would love some feedback on the shell script to make it as robust as possible, since its part of a deploy pipeline.
What's your opinion of breaking your pipeline with Sonar?

viesure/blog-sonar-build-breaker

Tests versus Documentation

Martin Huter — Sat, 16 Mar 2019 18:56:43 +0000

Have you ever been in the situation that you have been thrown under the bus and needed to fix a bug in an unknown code base with a real tight deadline? Or joined a new team/project which has tons of legacy code where you do not know where to start?

Well let's create some hypothetical project in our mind and we call it Schroedinger's repository. It has the described properties that its code base is rather large and you are not familiar with it. You remember that cat from school that was either dead or alive. Well in our example we won't murder innocent cats, but this project can have two representations:

You have a well-documented API and lots of comment flavor that is guiding you through the code, but tests are barely existent.
There is a good level of test coverage, tests are well structured in unit, integration, etc. tests. The downside here is that the API is lacking proper descriptions.

In my experience, if I need to fix a bug inside a code base I prefer proper automated tests, to see if I would break something. On the other hand, when my task is to enhance an API or create another layer upon something, proper documentation is preferably in my view.

Now I am asking you out there, if you would have to get in touch with our legacy Schroedinger's repository which state would you prefer, and what's your reasoning?

Another thing I want to point out:
We are all producing legacy code. The question is what should our legacy look like?

Contributing to OSS

Martin Huter — Wed, 06 Feb 2019 20:04:23 +0000

Hey guys out there! If you are like me you are using most of your time coding some bits of OSS. Have you ever thought about contributing to some of those nice frameworks, services and all this other cool stuff? Well I did, but I got to tell you I did not really achieve a lot.

I did some smaller pull requests on GitHub or tried to help to solve a issue with the investigation process. But that's it.

Whats your approach to contribute to OSS

I would like to know the stories of you guys and girls out there how you tackle this subject, or if you even tackle it? Have you any trick to select a good project that's worth your time?

How to improve my way to make a larger impact to the OSS world

Sooo, in future I want to sacrifice a hour every week to work on OSS, and try to report at dev.to about my progress, to have additional motivation to procrastinate this process anymore. This is about 0.5% of my time that should be doable.

Annex

First I wanted to thank @marek for his article about writing more, which motivated to put myself out there.

Three Arguments for Why You Should Write More

Marek Zaluski ・ Jan 28 '19 ・ 4 min read

#writing #career #discuss

Second I want to thank dev.to that there editor is in markdown, since I use this language on daily bases to write documentation, it was really easy to adjust.

Image origin: https://pixabay.com/

DEV Community: Martin Huter