DEV Community: Shmavon Gazanchyan

SonarQube and GitLab Setup

Shmavon Gazanchyan — Sun, 07 Jun 2020 22:39:37 +0000

SonarQube is a great tool for code quality and security checks. It is one of the tools we use to ensure quality of our products is measured and improved over time.

We also use GitLab – code management and CI/CD tool on some of our projects. SonarQube code analysis is integrated as a step in our GitLab CI pipelines.

For a long time I had a custom-built docker image for those purposes, automatically built and published to GitLab Container Registry.

Although quite automated, this custom-build docker image was adding to maintenance of the pipelines. I was happy to see that SonarSource has recently published official SonarScanner CLI docker image, and announced a support for GitLab integration in v8

SonarSource / sonar-scanner-cli-docker

Docker image for SonarScanner CLI

The docker image expects code being mounted on $PWD, which is set to /usr/src. This is different to where GitLab is mounting sources directory by default.

To make it all work the following environment variable needs to be set:

sonar:
  image: sonarsource/sonar-scanner-cli
  script:
    - sonar-scanner -Dsonar.qualitygate.wait=true
  allow_failure: true
  variables:
    SONAR_PROJECT_BASE_DIR: "${CI_PROJECT_DIR}"

Environment variable SONAR_PROJECT_BASE_DIR overrides default behavior. Predefined GitLab environment variable CI_PROJECT_DIR refers to the location where GitLab is mounting project source code.

Without this variable set the following misleading error will appear in GitLab log:

ERROR: Error during SonarScanner execution
java.lang.IllegalStateException: Unable to create user cache: /usr/src/.sonar/cache
    at org.sonarsource.scanner.api.internal.cache.FileCache.createDir(FileCache.java:147)
    at org.sonarsource.scanner.api.internal.cache.FileCache.<init>(FileCache.java:46)
    at org.sonarsource.scanner.api.internal.cache.FileCache.create(FileCache.java:52)
    at org.sonarsource.scanner.api.internal.cache.FileCacheBuilder.build(FileCacheBuilder.java:48)
    at org.sonarsource.scanner.api.internal.JarDownloaderFactory.create(JarDownloaderFactory.java:42)
    at org.sonarsource.scanner.api.internal.IsolatedLauncherFactory.createLauncher(IsolatedLauncherFactory.java:68)
    at org.sonarsource.scanner.api.EmbeddedScanner.doStart(EmbeddedScanner.java:185)
    at org.sonarsource.scanner.api.EmbeddedScanner.start(EmbeddedScanner.java:123)
    at org.sonarsource.scanner.cli.Main.execute(Main.java:73)
    at org.sonarsource.scanner.cli.Main.main(Main.java:61)
Caused by: java.nio.file.AccessDeniedException: /usr/src/.sonar
    at java.base/sun.nio.fs.UnixException.translateToIOException(Unknown Source)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source)
    at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(Unknown Source)
    at java.base/java.nio.file.Files.createDirectory(Unknown Source)
    at java.base/java.nio.file.Files.createAndCheckIsDirectory(Unknown Source)
    at java.base/java.nio.file.Files.createDirectories(Unknown Source)
    at org.sonarsource.scanner.api.internal.cache.FileCache.createDir(FileCache.java:145)
    ... 9 more
ERROR: 
ERROR: Re-run SonarScanner using the -X switch to enable full debug logging.

Improvements

It might be a good idea to cache Sonar Scanner directories to be re-used in future pipeline runs:

sonar:
  image: sonarsource/sonar-scanner-cli
  script:
    - sonar-scanner -Dsonar.qualitygate.wait=true
  allow_failure: true
  variables:
    SONAR_PROJECT_BASE_DIR: "${CI_PROJECT_DIR}"
  cache:
    key: "sonar-${CI_PROJECT_ID}"
    paths:
      - ".scannerwork"
      - ".sonar"

In my particular case, SonarQube analysis was not dependent on any other steps in the pipeline, so I've used needs and dependencies properties to make it run ahead of time and skip downloading artifacts of other steps.

sonar:
  image: sonarsource/sonar-scanner-cli
  needs: []
  dependencies: []
  script:
    - sonar-scanner -Dsonar.qualitygate.wait=true
  allow_failure: true
  variables:
    SONAR_PROJECT_BASE_DIR: "${CI_PROJECT_DIR}"
    GIT_DEPTH: 0
  cache:
    key: "sonar-${CI_PROJECT_ID}"
    paths:
      - ".scannerwork"
      - ".sonar"

Finally, I've used GIT_DEPTH environment variable to enforce Shallow Clone of our fairly large repository.

Hacktoberfest Tips

Shmavon Gazanchyan — Wed, 02 Oct 2019 16:15:53 +0000

October is here. For many GitHub repository maintainers (including me) it means the whole month of Pull Requests thanks to Hacktoberfest. This is the time when open source projects are getting a boost in contributions that help them develop new features, fix old bugs and attract new contributors.

If you plan to participate in this festival of PRs below is a couple of tips on how to make a good contribution.

Read the contributing guidelines

Many repositories will have a CONTRIBUTING.md file that contains important information on project code style, PR review workflow, test coverage requirements, and other contributing guidelines.

Note: the actual file name may vary: CONTRIBUTING, CONTRIBUTE, with or without .md extension, could be written in lowercase as well.

If you are struggling to find this document in the repository, GitHub has a convenient link to the file on Pull Requests page:

Give a clear description of your pull request

Help the maintainer quickly understand what problem your PR is solving. If there is an existing Issue on GitHub – reference it in the Pull Request description to give some more context.

This could be a very valuable contribution, but it is hard to say from the name or (absence of) description in the PR:

Here is a good example:

Match code style of the rest of the code

This is simple but often forgotten advice. Each project has its own code style and standards. When contributing to a project, make sure your code is following the same standards even if you don't agree with them.

If a project is using tabs for code indentation and you are a radical 4-space evangelist, when contributing to this project you are expected to follow their code style (yes, tabs) even if it is not mentioned in their contributing guidelines.

Code style guidelines are designed to make code look consistent – it greatly improves its readability.

Follow Pull Request template if it exists

GitHub provides a nice feature that allows maintainers to set a template for new Pull Requests. If the project you contributing to has such a template - read it carefully and follow steps described in it before submitting your PR.

Don't give up if maintainers do not respond instantly

Maintainers could be slow in response - unfortunately, it happens (I am guilty of this myself). Hacktoberfest is a busy period for all maintainers, so leave your PR open and move on - they will get back to you, just a little bit later.

Google Cloud App Engine Environment Variables

Shmavon Gazanchyan — Tue, 17 Sep 2019 16:41:50 +0000

One of my recent projects involved an application that had to be hosted on Google Cloud App Engine. I didn't use App Engine before, but I've managed apps on Heroku and OpenShift and was interested to see what Google Cloud PaaS had to offer.

It was a fairly standard Node.js application with most of the configuration done with environment variables. Soon it became clear that this could be a problem – App Engine does not support configurable environment variables.

Intentional or not, App Engine has only one way of defining those variables – in app.yaml configuration file. This file describes App Engine settings (runtime, url mappings etc.), including env_variables section that instructs App Engine to set environment variables on deployment.

Example app.yaml file content:



runtime: nodejs10

handlers:
- url: /api/.*
  script: auto
  secure: always

- url: /.*
  static_files: index.html
  upload: index.html
  secure: always
  http_headers:
    X-Frame-Options: deny
    X-DNS-Prefetch-Control: off
    X-XSS-Protection: 1; mode=block
    X-Permitted-Cross-Domain-Policies: none

env_variables:
    VAR1: 'VALUE1'
    VAR2: 'VALUE2'

Our deployment pipeline was already fully automated, so I needed to store app.yaml file somewhere to supply it to build server before pushing code to Google Cloud. Ideally, it would be application's code repository. However, having environment variables in app.yaml file caused an issue: we either needed to commit application configuration to the repository, or leave the whole file untracked. Neither of these options was suitable, so I started looking for any other ways of dealing with this App Engine limitation.

As a side note, Heroku and OpenShift (at least its previous incarnation) have an option to set environment variables from the web/command line interface, which simplified application configuration management.

My search brought me some disappointing results:

Store application configuration in the Google Cloud Datastore and read from it on application startup (link).
Encrypt configuration values with Cloud KMS and commit together with the rest of the app.yaml configuration (link).
Use separate app.yaml files for different environments (and, I guess, commit them all to the repository?) (same link).

Option #1 assumed additional component in our infrastructure and vendor lock-in to Google Cloud Datastore database, which was far from ideal.

Option #2 solved the security part of the problem, but would mean hardcoding encrypted environment-specific values to the codebase. It would also require to update the code with each new environment added or any changes to the existing environment variables. Not ideal.

Option #3 didn't solve he problem at all – code would still store information about its environments and application secrets will be available right in the code repository...

Extra step in deployment pipeline

Eventually, I've came up with an approach that involved compiling app.yaml file from template file during the build process. At that moment we used Google Cloud Build as a build server for CI/CD, but quickly moved to GitLab CI since Cloud Build does not support environment variables either.

To be fair, I should mention that Cloud Build supports "substitutions", which are remotely similar to environment variables. Unfortunately, the only way to pass substitutions to the build job is through command line arguments, which means managing environment variables somewhere outside. And this brings us back to the original problem...

The application was already using EJS library, so I used it to compile the template:



# app.tpl.yaml
runtime: nodejs10
env_variables:
  SSO_CLIENT_ID: <% SSO_CLIENT_ID %>
  SSO_SECRET: <% SSO_SECRET %>

with a script like that:



// bin/config-compile.js
const fs = require('fs');
const ejs = require('ejs');

const template = fs.readFileSync('app.tpl.yaml').toString();
const content = ejs.render(template, process.env);

fs.writeFileSync('app.yaml', content);

and GitLab CI step similar to this:



# .gitlab-ci.yml
config-compile:
  stage: build
  image: node:10
  script:
    - node bin/config-compile.js
  artifacts:
    paths:
      - app.yaml
    expire_in: 1 days
    when: always

This enabled us to manage application configuration in GitLab environment variables.

The approach can easily be adapted to any other templating library, programming language or build server, the code above is just an example.

A note on security

I found it interesting that one of the oldest Google Cloud products doesn't support such a common functionality. However, I accept there could be valid security reasons for doing so, e.g. exposure to dependency vulnerability similar to the one discovered in rest-client or typo-squatting like in npm registry. This is especially relevant as App Engine does not provide options to limit/manage outgoing connections from the environment.

Alternative solutions not covered in this post

"Secrets in Google App Engine" by Stuart Leitch
"How to use Environment Variables in GCloud App Engine" by Gunar Gessner