How I built my first bot using Typescript - Part #3: Code Coverage Reports

Hi everyone,

This the part #3 of the series, make sure to check the rest of them! Here's the previous episode.

Recently I've published a GitHub app. Darkest-PR is a bot for responding to the actions occurring in your repository by commenting them with quotes from the Darkest Dungeon game. The goal was to make development more fun and enjoyable thanks to the narrations by the Darkest-PR.

For this episode, I'll be covering code coverage reports, deployment and CI/CD operations of the app.

Episodes:

1. Beginning and architecture
2. Unit and feature testing, mocking & spying
3. Code coverage reports (you're here!)
4. Deployment and CI/CD
5. Publishing and final remarks

Don't forget to check the repository if you'd like to follow along!

Code Coverage

What is code coverage?

Code coverage is a metric that is used to measure the percentage and the sections of code that are subject to unit/feature testing. If any of your unit tests hits a certain line in a file at any point during testing, then that line has been successfully covered by tests.

Unit of code coverage is percentage (%). Higher it is, better it is. Because higher code coverage means sufficient amount of tests are prepared to cover those sections of the code. By my experience, for commercial applications your code coverage should be at least 50% , that is the vital minimum amount, anything below that means you're shooting in the dark and may encounter bugs at any time. For open-source projects and especially libraries, you should aim for around 95% coverage, 90% as minimum. Because libraries are meant to be used by hundreds if not thousands of projects, so if anything goes wrong in your library it will affects all those projects, teams and their users. So be a responsible open-source developer and prepare sufficient unit tests to reach at least %90 coverage.

In theory; higher the code coverage amount, unlikely you're to encounter unexpected bugs. While high code coverage doesn’t guarantee bug-free code, it does give you confidence that a significant portion of your code is being tested, helping to catch potential issues early in the development process.

Types of Code Coverage

Before diving into the specifics of how we handle code coverage in Darkest-PR, let’s quickly go over the different types of code coverage:

• Line Coverage: Measures the percentage of lines of code that have been executed during tests.
• Branch Coverage: Focuses on the decision points in your code, ensuring that every possible branch (e.g., if/else statements) is tested.
• Function Coverage: Tracks whether each function in your codebase has been called during the testing process.
• Statement Coverage: Similar to line coverage but focuses on individual statements rather than lines.

How to generate code coverage reports

There are various methods and frameworks for generating code coverage reports, it all depends on your tech stack. Some famous code coverage report generation tools:

• V8

  • Javascript/Node.js
  • Node.js built in test coverage engine
  • Node.js has built-in V8 code coverage support, which generates detailed code coverage reports in JSON format. These reports can be converted to more readable formats (e.g., HTML) using tools like c8 or v8-to-istanbul.

• Istanbul.js

  • JavaScript/TypeScript
  • Provides code coverage reports for JavaScript applications, directly generating detailed HTML, JSON, and text reports.

• PHPUnit

  • PHP
  • A testing framework for PHP that also generates code coverage reports in formats like HTML and XML.

• Clover

  • Java/Groovy
  • Generates detailed code coverage reports for Java and Groovy projects, with output in HTML, XML, and JSON.

And the list goes on. For each language there are dozens of test coverage engines to choose from. You should evaluate each one's pros and cons and deduce it per your requirements.

For this project I went along with V8. I tried Istanbul engine too but I liked the performance and metrics of V8 more. Here's what output Vitest with V8 provides us in CLI:

In the output above, we can clearly see which file has what percentage of coverage, and which files are lacking code coverage.

When you run vitest --coverage it will automatically create a /coverage folder in your repository's root. This folder contains your coverage reports in various formats. XML, JSON and HTML. For demonstration purposes you may open the index.html file in a browser to see the report yourself.

You can also try vitest --ui for interactive code coverage report webpage.

Each of these code coverage reports (XML, JSON, HTML) contains the same coverage analysis but in different formats. In the next title, we'll explore how we can use these reports to publish them publicly and consistently.

SaaS tool for publishing coverage reports for free

After being able to generate code coverage reports, now we can move on to publishing them on an online platform so these reports will be accessible anywhere anytime.

CodeCov is a SaaS tool for uploading and visualizing your code coverage reports. It is free for open-source projects. It enables visually compelling charts to be displayed for your reports.

Moreover, CodeCov also offers support for Markdown badges so you can easily embed your coverage percentage to your repository.

Simply login with your GitHub profile and add your repository to CodeCov. Afterwards you need to automatically relay code coverage reports whenever it is generated. Now you might ask: "Why, how am I supposed to send it to CodeCov if I generate it locally?" That is simple, you need to setup a CI/CD pipeline to:

1. Clone project
2. Install dependencies
3. Build the project
4. Run tests and & generate code coverage reports
5. Send generated coverage reports to CodeCov

CodeCov has a GitHub action to easily send reports, it is called codecov/codecov-action.

Here's the CI/CD action file for the project which reflects the steps mentioned above:

name: Node.js CI

on:
  push:
    branches: [ "master" ]
  pull_request:
    branches: [ "master" ]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [18.x]

    steps:
    - uses: actions/checkout@v4
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v3
      with:
        node-version: ${{ matrix.node-version }}
        cache: 'npm'
    - run: npm ci
    - run: npm run build
    - run: npm test -- --coverage
    - name: Upload coverage reports to Codecov
      uses: codecov/codecov-action@v4.0.1
      with:
       token: ${{ secrets.CODECOV_TOKEN }} #add your CodeCov secret to repository secrets!

And after this, whenever a push occurs or a PR is created for master; our code coverage reports will be submitted to CodeCov automatically and the metrics will always be up-to-date.