Generating Code Coverage from the Continuous Integration Process

In the previous article we started working with the CircleCI platform, by setting up testing infrastructure for our code.

In this article, we are going to set up code coverage for the project.

The Plan

• Instruct test runner (jest) to generate and save code coverage files for our tests.
• Upload those reports to external service for aggregation and display. For that, we are going to use codecov.io for code coverage.

First, you will need to go to codecov.io to create an account and chose the repository you want to show code coverage for. Then you need to install codecov.io application from the github marketplace.

As a reminder, this is the CircleCI configuration file from the previous article.

version: 2.1
jobs:
  node-v10:
    docker:
      - image: circleci/node:10
    steps:
      - test
  node-v12:
    docker:
      - image: circleci/node:12
    steps:
      - test
  node-v13:
    docker:
      - image: circleci/node:13
    steps:
      - test
commands:
  test:
    steps:
      - checkout
      - restore_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
      - run:
          name: install-dependancies
          command: npm ci
      - save_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - $HOME/.npm
      - run:
          name: unit test
          command: |
            npm run ci:test
workflows:
  version: 2
  build_and_test:
    jobs:
      - node-v10
      - node-v12
      - node-v13

We are going to continue to build upon this file by adding code coverage functionality.

How is Code Coverage Generated

Since we are using Jest for testing the code, there is not much that we need to do to set up code coverage, because jest comes with code coverage out of the box.

Next, we need to modify ci:test npm script inside the package.json

Our previous npm script for running jest tests was:

"ci:test": "jest --runInBand --ci"

We are going to change the script slightly so we can instruct jest to also generate code coverage files and save them in the coverage directory in the root of the project.

"ci:test": "jest --runInBand --ci  --reporters=default  --coverage --coverageDirectory=coverage"

Setting up Code Coverage on the CircleCI platform

Setting up code coverage is fairly easy. After jest tests the code and generates code coverage files we need to save them somewhere and then upload them to the codecov.io
In the previous article we talked about how code is running inside Linux containers and how they are ephemeral and are discarded as soon as the jobs that are running inside them are done, so we are going to need to take the files that are generated by jest and save them somewhere outside of the containers before the containers are destroyed.

Then, after we save the files somewhere outside the containers, the only step left is to upload those files to the codecov.io service.

To use the codecov.io service via the circleCI platform, we are going to use the official codecov.io "orb".

CircleCI Orbs Concept Explained

CircleCI platform has a concept of orbs,
from the docs:

Orbs are reusable packages of CircleCI configuration that you may share across projects, enabling you to create encapsulated, parameterized commands, jobs, and executors that can be used across multiple projects.

You can consider them as classes in object oriented programming. A blueprint for doing certain tasks. Orbs simplify tasks by hiding all necessary steps that need to be done and only exposing a few methods to get the tasks going.

You can take a look at the actual code for the codecov orb to see how much typing using the orbs saves us. Also when using officially supported orbs you can be sure that your CircleCI jobs will work as expected when integrating with the external services. Many online services that integrate with CircleCI have their own orbs that make it easy to integrate with the continuous integration process.

To use any orb we need to declare it under the orbs key, so let's add the orbs key to the config.yml file.

version: 2.1
orbs:
  codecov: codecov/codecov@1.0.5
jobs:

Next, we are going to add additional steps to be executed after the code is successfully tested.

jobs:
  # ... node-v10 job
  node-v12:
  docker:
    - image: circleci/node:12
  steps:
    - test # test the code
    - store-coverage-data # store coverage files outside of the container command
    - upload-coverage # upload files to the codecov.io command
  # ... node-v13 job

You might notice that in the previous example I have omitted the code for the node-v10 and node-v13 jobs. The reason for this is that I have decided to only use code coverage from one job/container (node-v12). Coverage data from the node-v10 and node-v13 jobs will be ignored, Jest will create code coverage but it will not be used by the codecov.io orb (we could further optimize those jobs to not generate code coverage at all but that is beyond the scope of this article).

Code Coverage Commands

Now let's take a look at the commands that are used in the steps above.

commands:
  store-coverage-data:
    steps:
      - store_artifacts:
          path: coverage
  upload-coverage:
    steps:
      - codecov/upload:
          file: coverage/coverage-final.json

• step: store-coverage-data

  • store_artifacts - This is a special built-in command in the CircleCI platform that can save anything inside the container to the CircleCI permanent storage (move it outside of the container). We are using it to save our coverage files. From the docs:

  Artifacts persist data after a job is completed and may be used for longer-term storage of the outputs of your build process. For example, when a Java build/test process finishes, the output of the process is saved as a .jar file. CircleCI can store this file as an artifact, keeping it available long after the process has finished.

  • path This is the path inside the container that we want to be saved externally. Please note that we are using a relative path to the coverage directory inside our working directory (root directory of our checked out code)

• step: upload-coverage

  • codecov/upload: This is a custom command (upload) defined via codecov orb. As you can see you can treat it as a class instance method codecov->upload This command uploads required files for the code coverage service.
  • file path to the file we want to upload.

Using Jest-Junit for Test Results

There is one additional step that we can add when setting up code coverage. We can store test performance results for later review.
Inside the circleCI UI, there can be an additional tab that can show you which tests are failing and which tests took a long time to run.
To enable this functionality, we need to install additional node module jest-junit and configure it via package.json

"jest-junit": {
  "outputDirectory": "./reports/junit",
  "outputName": "test-results.xml"
}

once again, we need to modify ci:test npm script inside the package.json

"ci:test": "jest --runInBand --ci  --reporters=default --reporters=jest-junit --coverage --coverageDirectory=coverage"

Next we need to modify circleCI configuration file and add another step to the test command: store_test_results
From the docs

store_test_results - Special step used to upload and store test results for a build. Test results are visible on the CircleCI web application, under each build’s “Test Summary” section. Storing test results is useful for timing analysis of your test suites.

modified configuration file:

commands:
  test:
    steps:
      - checkout
      - restore_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
      - run:
          name: install-dependancies
          command: npm ci
      - save_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - $HOME/.npm
      - run:
          name: unit test
          command: |
            mkdir -p ./reports/junit
            npm run ci:test
      - store_test_results: #<--- store test results for CircleCI
          path: ./reports/junit/

And that's it, that is all it takes to set up code coverage via the CircleCI platform.

Complete config file:

version: 2.1
orbs:
  codecov: codecov/codecov@1.0.5
jobs:
  node-v10:
    docker:
      - image: circleci/node:10
    steps:
      - test
  node-v12:
    docker:
      - image: circleci/node:12
    steps:
      - test
      - store-coverage-data
      - upload-coverage
  node-v13:
    docker:
      - image: circleci/node:13
    steps:
      - test
commands:
  store-coverage-data:
    steps:
      - store_artifacts:
          path: coverage
  test:
    steps:
      - checkout
      - restore_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
      - run:
          name: install-dependancies
          command: npm ci
      - save_cache:
          key: dependency-cache-{{ checksum "package-lock.json" }}
          paths:
            - $HOME/.npm
      - run:
          name: unit test
          command: |
            npm run ci:test
      - store_test_results:
          path: ./reports/junit/
  upload-coverage:
    steps:
      - codecov/upload:
          file: coverage/coverage-final.json
workflows:
  version: 2
  build_and_test:
    jobs:
      - node-v10
      - node-v12
      - node-v13

Bonus: Configuring Jest to Fail

You can also set up Jest to fail the test if code coverage stats are below a certain threshold. You can do it on a global level or per directory/file.

This goes into your package.json file.

"jest":{
  "coverageThreshold": {
    "global": {
      "branches": 50,
      "functions": 50,
      "lines": 50,
      "statements": 50
    },
    "./src/components/": {
      "branches": 40,
      "statements": 40
    },
  }
}

You can read more about jest.io coverage threshold here

To use code coverage with private repositories, you will need to set up a $CODECOV_TOKEN environment token via CircleCI dashboard
You can obtain the token when you connect your repository with codecov.io via their website.

Comments

[Francesco Bianco]

Please take a look at LCOV.SH full BASH implementation of coverage… no need for additional interpreters like RUBY or BINARY executable in your machine. Check coverage of BASH just with BASH github.com/javanile/lcov.sh