PCCR - Perfect Code Coverage Reporting

Cover photo by Jonathan Cooper

---

If your project has perfect code coverage, and no one, not even you, knows about it, do doves cry?

We can never achieve perfection, but if we limit scope it is attainable. ;)

Perfect code coverage means 100%, including branches. NOTE: Easiest to accomplish when code has no branches!

Perfect code coverage reporting would be to tell everyone about it, badge it up, and leave comments about it in PRs.

How do you get there?

---

I recently wrote a gem (i.e. a Ruby library) that provides a canonical regex matching the set of unicode Gitmoji characters.

A simple library, with no branch statements, was easy to get coverage perfection, but a bit complicated to get reporting perfection...

My example utilizes Ruby, as my language of choice, but most of this, and particularly the Github Actions Coverage Workflow, will apply to any language.

---

Benefits

README.md

Self-promotion in open source projects is important.

In the README.md now there are glorious badges.

Pull Request Automation

Additionally, PRs to the project should get a comment like the below from the CodeCoverageSummary Github Action

Package	Line Rate	Health
gitmoji-regex	100%	✔
Summary	100% (50 / 50)	✔

---

Tools

• CodeClimate SAAS
• CodeCov SAAS
• Coveralls SAAS
• irongut/CodeCoverageSummary Github Action
• amancevice/setup-code-climate Github Action
• Coverage-related Gems I use:
  • gem "codecov", "~> 0.6" - Note: codecov/codecov-action Github Action is an alternative for those not using Ruby. I left an example in the coverage workflow
  • gem "simplecov", "~> 0.21", require: false
  • gem "simplecov-cobertura" provides XML format for Jenkins, or other tools
  • gem 'simplecov-json' provides JSON format for CodeClimate
  • gem "simplecov-lcov", "~> 0.8", require: false provides lcov format

---

Implementation

Coverage Workflow

This is the meat and potatoes that pulls everything together!

name: Code Coverage

env:
  CI_CODECOV: true
  COVER_ALL: true

on:
  push:
    branches:
      - 'main'
      - '*-maintenance'
      - '*-dev'
      - '*-stable'
    tags:
      - '!*' # Do not execute on tags
  pull_request:
    branches:
      - '*'
  # Allow manually triggering the workflow.
  workflow_dispatch:

# Cancels all previous workflow runs for the same branch that have not yet completed.
concurrency:
  # The concurrency group contains the workflow name and the branch name.
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    name: Specs with Coverage - Ruby ${{ matrix.ruby }} ${{ matrix.name_extra || '' }}
    if: "!contains(github.event.commits[0].message, '[ci skip]') && !contains(github.event.commits[0].message, '[skip ci]')"
    strategy:
      fail-fast: false
      matrix:
        experimental: [false]
        rubygems:
          - latest
        bundler:
          - latest
        ruby:
          - "2.7"

    runs-on: ubuntu-latest
    continue-on-error: ${{ matrix.experimental || endsWith(matrix.ruby, 'head') }}
    steps:
      - uses: amancevice/setup-code-climate@v0
        name: CodeClimate Install
        if: matrix.ruby == '2.7' && github.event_name != 'pull_request' && always()
        with:
          cc_test_reporter_id: ${{ secrets.CC_TEST_REPORTER_ID }}

      - name: Checkout
        uses: actions/checkout@v3

      - name: Setup Ruby & Bundle
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: ${{ matrix.ruby }}
          rubygems: ${{ matrix.rubygems }}
          bundler: ${{ matrix.bundler }}
          bundler-cache: true

      - name: CodeClimate Pre-build Notification
        run: cc-test-reporter before-build
        if: matrix.ruby == '2.7' && github.event_name != 'pull_request' && always()
        continue-on-error: ${{ matrix.experimental != 'false' }}

      - name: Run tests
        run: bundle exec rake test

      - name: CodeClimate Post-build Notification
        run: cc-test-reporter after-build
        if: matrix.ruby == '2.7' && github.event_name != 'pull_request' && always()
        continue-on-error: ${{ matrix.experimental != 'false' }}

      - name: Code Coverage Summary Report
        uses: irongut/CodeCoverageSummary@v1.2.0
        with:
          filename: ./coverage/coverage.xml
          badge: true
          fail_below_min: true
          format: markdown
          hide_branch_rate: true
          hide_complexity: true
          indicators: true
          output: both
          thresholds: '95 97'
        continue-on-error: ${{ matrix.experimental != 'false' }}

      - name: Add Coverage PR Comment
        uses: marocchino/sticky-pull-request-comment@v2
        if: matrix.ruby == '2.7' && always()
        with:
          recreate: true
          path: code-coverage-results.md
        continue-on-error: ${{ matrix.experimental != 'false' }}

      - name: Coveralls
        uses: coverallsapp/github-action@master
        if: matrix.ruby == '2.7' && github.event_name != 'pull_request' && always()
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
        continue-on-error: ${{ matrix.experimental != 'false' }}

#      Using the codecov gem instead.
#      - name: CodeCov
#        uses: codecov/codecov-action@v2
#        if: matrix.ruby == '2.7' && github.event_name != 'pull_request' && always()
#        with:
#          files: ./coverage/coverage.xml
#          flags: unittests
#          name: codecov-upload
#          fail_ci_if_error: true
#        continue-on-error: ${{ matrix.experimental != 'false' }}

Simplecov

CONSTANTS NOTE: Some constants being used below, e.g. RUN_COVERAGE and ALL_FORMATTERS, are set in the spec/spec_helper.rb, which is run before the .simplecov config is loaded. Others, e.g. RUBY_ENGINE, RUBY_VERSION, are standard Ruby.

You must require "simplecov" before loading the code to be coverage checked.

if RUN_COVERAGE
  require "simplecov" # Config file `.simplecov` is run immediately when simplecov loads
  require "codecov"
  require "simplecov-json"
  require "simplecov-lcov"
  require "simplecov-cobertura"
  # This will override any formatter set in .simplecov
  if ALL_FORMATTERS
    SimpleCov::Formatter::LcovFormatter.config do |c|
      c.report_with_single_file = true
      c.single_report_path = "coverage/lcov.info"
    end

    SimpleCov.formatters = [
      SimpleCov::Formatter::HTMLFormatter,
      SimpleCov::Formatter::CoberturaFormatter,
      SimpleCov::Formatter::LcovFormatter,
      SimpleCov::Formatter::JSONFormatter, # For CodeClimate
      SimpleCov::Formatter::Codecov # For CodeCov
    ]
  end
end

Configure it with a .simplecov config as below:

# frozen_string_literal: true

# To get coverage
# On Local, default (HTML) output, it just works, coverage is turned on:
#   bundle exec rspec spec
# On Local, all output formats:
#   COVER_ALL=true bundle exec rspec spec
#
# On CI, all output formats, the ENV variables CI is always set,
#   and COVER_ALL, and CI_CODECOV, are set in the coverage.yml workflow only,
#   so coverage only runs in that workflow, and outputs all formats.
#

if RUN_COVERAGE
  SimpleCov.start do
    enable_coverage :branch
    primary_coverage :branch
    add_filter "test"
    track_files "**/*.rb"

    if ALL_FORMATTERS
      command_name "#{ENV["GITHUB_WORKFLOW"]} Job #{ENV["GITHUB_RUN_ID"]}:#{ENV["GITHUB_RUN_NUMBER"]}"
    else
      formatter SimpleCov::Formatter::HTMLFormatter
    end

    minimum_coverage(100)
  end
else
  puts "Not running coverage on #{RUBY_ENGINE} #{RUBY_VERSION}"
end

---

End Result

Github Actions Run

I have since re-implemented this pattern in other gems, including the oauth gem, which uses MiniTest instead of RSpec, and which does not have 100% test coverage (yet). Most of the implementation is the same.

Comments

[CoverInfo.xyz]

Need cover art for your next release? Learn all the correct album cover art sizes, dimensions and format required for digital stores like Spotify and Apple.