For my projects, I like to use TravisCI for running my builds and tests. If I'm working with a language that supports it, I also like to publish code coverage results from the test run to coveralls.io. A little while back I found a cross-platform tool to calculate coverage results for .NET Core projects:
lucaslorentz / minicover
Cross platform code coverage tool for .NET Core
MiniCover
Code Coverage Tool for .NET Core
Supported .NET Core SDKs
- 2.1 (Global tool)
- 2.2 (Global tool)
- 3.0 (Global tool or local tool)
- 3.1 (Global tool or local tool)
Installation
MiniCover can be installed as a global tool:
dotnet tool install --global minicover
Or local tool:
dotnet tool install minicover
Commands
This is a simplified documentation of MiniCover commands and options.
Use --help
for more information:
minicover --help
When installed as local tool, MiniCover commands must be prefixed with dotnet
. Example:
dotnet minicover --help
Instrument
minicover instrument
Use this command to instrument assemblies to record code coverage.
It is based on the following main options:
option | description | type | default |
---|---|---|---|
sources | source files to track coverage | glob | src/**/*.cs |
exclude-sources | exceptions to source option | glob |
**/bin/**/*.cs and **/obj/**/*.cs
|
tests | test files used to recognize test methods | glob |
tests/**/*.cs and test/**/*.cs
|
exclude-tests | exceptions to tests option | glob |
**/bin/**/*.cs and **/obj/**/*.cs
|
assemblies | assemblies |
All of the example code for this post is available on my GitHub profile:
nlowe / coveralls-netcore
coveralls.io example for a dotnet core app
Coveralls Netcore Sample
Getting code coverage into coveralls.io from a travis build of a dotnet core project.
Coverage generated using MiniCover via Cake.MiniCover
Using MiniCover
MiniCover is distributed as a dotnet
cli tool. We either need to add it as a DotNetCliToolReference
to one of the top-level projects in the repo or as a dedicated tools project. I like to place the reference in ./minicover/minicover.csproj
for reasons that will become clear soon:
<!-- ./minicover/minicover.csproj -->
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp2.0</TargetFramework>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<DotNetCliToolReference Include="MiniCover" Version="2.0.0-ci-20180517205544" />
</ItemGroup>
</Project>
Be sure to run dotnet restore
on this project.
This is required until MiniCover is updated to support installation as a global tool (added in .NET Core 2.1):
Publish as .NET Core Global Tool #90
With the upcoming release of dotnet core 2.1 they're introducing global tools. It works like NPM global tools (npm install -g package)
That would a nice alternative way to install MiniCover. So instead of having a tools project you can simply do dotnet install tool -g minicover
Since it's still being actively developed, MiniCover is distributed as prerelease packages. Check the nuget feed to get the latest version.
Generating Coverage
We're ready to run our tests with coverage:
Build the project
Newer versions of .NET Core include an implicit restore here. If you're using an old version of .NET Core be sure to do a dotnet restore
first.
dotnet build
Instrument the assemblies under test.
This inserts IL instructions into your assemblies to write coverage to a results file:
cd minicover
dotnet minicover instrument --workdir ../ --assemblies test/**/bin/**/*.dll --sources src/**/*.cs
dotnet minocover reset
cd ..
Run your tests:
You have to tell dotnet
not to rebuild assemblies when running tests. Otherwise the instrumentation we just added will be overwritten and no coverage will be tracked.
for project in test/**/*.csproj; do dotnet test --no-build $project; done
Generate Reports:
MiniCover includes support for rendering many types of reports. The most basic type is a Console report with a fatal threshold. If coverage is below this threshold, it will exit with a non-zero exit code allowing you to fail the build if coverage is too low.
cd minicover
dotnet minicover report --workdir ../ --threshold 90
cd ..
Here's a sample of what this looks like (In a TTY that supports colors, files above the threshold will be green and files below the threshold will be red):
+-------------------+-------+---------------+------------+
| File | Lines | Covered Lines | Percentage |
+-------------------+-------+---------------+------------+
| src/MyLib/Math.cs | 2 | 1 | 50.000% |
+-------------------+-------+---------------+------------+
| All files | 2 | 1 | 50.000% |
+-------------------+-------+---------------+------------+
Uninstrument Assemblies
If you are planning on publishing artifacts for this build, be sure to uninstrument assemblies:
cd minicover
dotnet minicover uninstrument --workdir ../
cd ..
Simplify things with Cake
While this is a great way to get code coverage results, it is a little tedious. For my .NET Projects, I prefer to use cake
to script the build process:
cake-build / cake
Cake (C# Make) is a cross platform build automation system.
Cake
Cake (C# Make) is a build automation system with a C# DSL to do things like compiling code, copy files/folders, running unit tests, compress files and build NuGet packages.
Continuous integration
Code Coverage
Table of Contents
Documentation
You can read the latest documentation at https://cakebuild.net/.
Example
This example downloads the Cake bootstrapper and executes a simple build script The bootstrapper is used to bootstrap Cake in a simple way and is not in required in any way to execute build scripts. If you…
To get started with cake, you will need two files, a build.cake
build script and a platform-specific bootstrapper script. Since we can build and generate coverage for .NET Core projects on Windows, macOS, and Linux, we can include a PowerShell script for Windows and a Bash Script for the other platforms. If you want to get started with building a .NET Core Project with cake, feel free to use my bootstrapper scripts:
nlowe / cake-bootstrap-dotnet
Bootstrapper scripts for using cake with dotnet core
Cake Bootstrapper for dotnet core projects
Bootstrap cake for dotnet core projects without needing to install mono. Options (environment variables):
-
TOOLS_DIR
: The path to install cake tools to../tools
by default -
CAKE_VERSION
: The version of cake to install.0.26.1
by default. To upgrade cake, delete yourTOOLS_DIR
and change this variable. -
CAKE_NETCOREAPP_VERSION
: Thenetcoreapp
version to use for the tools dummy project.2.0
by default. Must be compatible withCake.CoreCLR
All other options are present as with the standard bootstrap scripts.
Cake is extremely extensible via the addin
system. This restores extensions to Cake as a nuget
package before running the build script. I've made a Cake addin for minicover that greatly simplifies the coverage generation process:
nlowe / Cake.MiniCover
A Cake Addin for Minicover, making it as easy as possible to get cross-platform code coverage on dotnet core
Cake.MiniCover
Usage
Until lucaslorentz/minicover#31 is
resolved, you need to call the SetMiniCoverToolsProject
alias to locate the tools project:
#addin "Cake.MiniCover"
SetMiniCoverToolsProject("./minicover/minicover.csproj")
// ...
Task("Coverage")
.IsDependentOn("build")
.Does(() =>
{
MiniCover(tool =>
{
foreach(var project in GetFiles("./test/**/*.csproj"))
{
tool.DotNetCoreTest(project.FullPath, new DotNetCoreTestSettings()
{
// Required to keep instrumentation added by MiniCover
NoBuild = true,
Configuration = configuration
});
}
},
new MiniCoverSettings()
.WithAssembliesMatching("./test/**/*.dll")
.WithSourcesMatching("./src/**/*.cs")
.GenerateReport(ReportType.CONSOLE | ReportType.XML)
);
});
// ...
If you need more fine-graned control or have multiple test targets, you can call the aliases individually:
#addin "Cake.MiniCover"
SetMiniCoverToolsProject
…The Build Script
Cake organizes the build script as a series of Task
s with dependencies. Addins expose aliases for us to call.
Referencing the Addin
The first order of business is referencing our addin:
#addin "nuget:?package=Cake.MiniCover&version=0.29.0-next20180721071547&prerelease"
Note: I recently fixed support for publishing coverage to https://coveralls.io. For now, you will need to reference the
0.29.0-next20180721071547
pre-release ofCake.MiniCover
until0.29.0
is released.Additionally, this version of
Cake.MiniCover
requiresCake
0.29+. You will need to use the bootstrap scripts from the example repo I linked at the beginning of this post or update them to referenceCake
0.29+
Locating the Tools Project
Next, we need to tell Cake where our MiniCover tools project is located:
SetMiniCoverToolsProject("./minicover/minicover.csproj");
Writing the Test Target
We can wrap our DotNetCoreTest
call in a MiniCover
call. This will automatically instrument and uninstrument assemblies and run any reports we request:
Task("Test")
.IsDependentOn("Build")
.Does(() =>
{
MiniCover(tool =>
{
foreach(var project in GetFiles("./test/**/*.csproj"))
{
DotNetCoreTest(project.FullPath, new DotNetCoreTestSettings
{
Configuration = configuration,
NoRestore = true,
NoBuild = true
});
}
},
new MiniCoverSettings()
.WithAssembliesMatching("./test/**/*.dll")
.WithSourcesMatching("./src/**/*.cs")
.WithNonFatalThreshold()
.GenerateReport(ReportType.CONSOLE)
);
});
For brevity I've only included the Test
task here. Please see the full build script for the Build
task definition and other variables required to execute this task.
Note the WithNonFatalThreshold()
setting. This will disable exiting with an error if coverage is below the specified threshold, which is useful if you want your build system to track builds but have a different system for tracking coverage.
Generating Coverage Results
Now, we can simply invoke our build script to run our unit tests and print a coverage report to the console:
# On macOS / Linux:
./build.sh -t test
Or
# Windows:
./build.ps1 -t Build
Cake will figure out the task dependencies and execute them in the proper order, showing output and keeping track of how long each task takes to run:
$ ./build.sh -t test
Installing Cake 0.29.0
Writing /tmp/tmpcaiVOk.tmp
info : Adding PackageReference for package 'Cake.CoreCLR' into project '/home/nathan/projects/coveralls-netcore/tools/cake.csproj'.
log : Restoring packages for /home/nathan/projects/coveralls-netcore/tools/cake.csproj...
info : GET https://api.nuget.org/v3-flatcontainer/cake.coreclr/index.json
info : OK https://api.nuget.org/v3-flatcontainer/cake.coreclr/index.json 135ms
info : GET https://api.nuget.org/v3-flatcontainer/cake.coreclr/0.29.0/cake.coreclr.0.29.0.nupkg
info : OK https://api.nuget.org/v3-flatcontainer/cake.coreclr/0.29.0/cake.coreclr.0.29.0.nupkg 49ms
log : Installing Cake.CoreCLR 0.29.0.
info : Package 'Cake.CoreCLR' is compatible with all the specified frameworks in project '/home/nathan/projects/coveralls-netcore/tools/cake.csproj'.
info : PackageReference for package 'Cake.CoreCLR' version '0.29.0' added to file '/home/nathan/projects/coveralls-netcore/tools/cake.csproj'.
Analyzing build script...
Processing build script...
Installing addins...
Compiling build script...
Restoring packages for /home/nathan/projects/coveralls-netcore/minicover/minicover.csproj...
Restore completed in 73.51 ms for /home/nathan/projects/coveralls-netcore/minicover/minicover.csproj.
Generating MSBuild file /home/nathan/projects/coveralls-netcore/minicover/obj/minicover.csproj.nuget.g.props.
Generating MSBuild file /home/nathan/projects/coveralls-netcore/minicover/obj/minicover.csproj.nuget.g.targets.
Restore completed in 180.67 ms for /home/nathan/projects/coveralls-netcore/minicover/minicover.csproj.
========================================
Restore
========================================
Executing task: Restore
Restoring packages for /home/nathan/projects/coveralls-netcore/test/MyLib.Tests/MyLib.Tests.csproj...
Restoring packages for /home/nathan/projects/coveralls-netcore/src/MyLib/MyLib.csproj...
Restore completed in 63.88 ms for /home/nathan/projects/coveralls-netcore/test/MyLib.Tests/MyLib.Tests.csproj.
/home/nathan/projects/coveralls-netcore/src/MyLib/MyLib.csproj : warning NU1603: MyLib depends on NETStandard.Library (>= 2.0.2-servicing-26420-0) but NETStandard.Library 2.0.2-servicing-26420-0 was not found. An approximate best match of NETStandard.Library 2.0.2 was resolved. [/home/nathan/projects/coveralls-netcore/coveralls-netcore.sln]
Generating MSBuild file /home/nathan/projects/coveralls-netcore/test/MyLib.Tests/obj/MyLib.Tests.csproj.nuget.g.props.
Generating MSBuild file /home/nathan/projects/coveralls-netcore/src/MyLib/obj/MyLib.csproj.nuget.g.props.
Generating MSBuild file /home/nathan/projects/coveralls-netcore/test/MyLib.Tests/obj/MyLib.Tests.csproj.nuget.g.targets.
Generating MSBuild file /home/nathan/projects/coveralls-netcore/src/MyLib/obj/MyLib.csproj.nuget.g.targets.
Restore completed in 655.78 ms for /home/nathan/projects/coveralls-netcore/src/MyLib/MyLib.csproj.
Restore completed in 655.63 ms for /home/nathan/projects/coveralls-netcore/test/MyLib.Tests/MyLib.Tests.csproj.
Finished executing task: Restore
========================================
Build
========================================
Executing task: Build
Microsoft (R) Build Engine version 15.7.179.62826 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.
/home/nathan/projects/coveralls-netcore/src/MyLib/MyLib.csproj : warning NU1603: MyLib depends on NETStandard.Library (>= 2.0.2-servicing-26420-0) but NETStandard.Library 2.0.2-servicing-26420-0 was not found. An approximate best match of NETStandard.Library 2.0.2 was resolved.
MyLib -> /home/nathan/projects/coveralls-netcore/src/MyLib/bin/Release/netstandard2.0/MyLib.dll
MyLib.Tests -> /home/nathan/projects/coveralls-netcore/test/MyLib.Tests/bin/Release/netcoreapp2.1/MyLib.Tests.dll
Build succeeded.
/home/nathan/projects/coveralls-netcore/src/MyLib/MyLib.csproj : warning NU1603: MyLib depends on NETStandard.Library (>= 2.0.2-servicing-26420-0) but NETStandard.Library 2.0.2-servicing-26420-0 was not found. An approximate best match of NETStandard.Library 2.0.2 was resolved.
1 Warning(s)
0 Error(s)
Time Elapsed 00:00:02.64
Finished executing task: Build
========================================
Test
========================================
Executing task: Test
Assembly resolver search directories:
/home/nathan/projects/coveralls-netcore/test/MyLib.Tests/obj/Release/netcoreapp2.1
Assembly resolver search directories:
/home/nathan/projects/coveralls-netcore/test/MyLib.Tests/bin/Release/netcoreapp2.1
/home/nathan/.nuget/packages
/opt/dotnet/sdk/NuGetFallbackFolder
Instrumenting assembly "MyLib"
Changing working directory to '/home/nathan/projects/coveralls-netcore'
Reset coverage for directory: '/home/nathan/projects/coveralls-netcore' on pattern './coverage-hits.txt'
Directory is already cleared
Test run for /home/nathan/projects/coveralls-netcore/test/MyLib.Tests/bin/Release/netcoreapp2.1/MyLib.Tests.dll(.NETCoreApp,Version=v2.1)
Microsoft (R) Test Execution Command Line Tool Version 15.3.0-preview-20170628-02
Copyright (c) Microsoft Corporation. All rights reserved.
Starting test execution, please wait...
[xUnit.net 00:00:00.4964089] Discovering: MyLib.Tests
[xUnit.net 00:00:00.5609486] Discovered: MyLib.Tests
[xUnit.net 00:00:00.5680125] Starting: MyLib.Tests
[xUnit.net 00:00:00.7413388] Finished: MyLib.Tests
Total tests: 1. Passed: 1. Failed: 0. Skipped: 0.
Test Run Successful.
Test execution time: 1.6506 Seconds
Changing working directory to '/home/nathan/projects/coveralls-netcore'
+-------------------+-------+---------------+------------+
| File | Lines | Covered Lines | Percentage |
+-------------------+-------+---------------+------------+
| src/MyLib/Math.cs | 2 | 1 | 50.000% |
+-------------------+-------+---------------+------------+
| All files | 2 | 1 | 50.000% |
+-------------------+-------+---------------+------------+
Finished executing task: Test
Task Duration
--------------------------------------------------
Restore 00:00:01.9525117
Build 00:00:02.9521372
Test 00:00:06.5041439
--------------------------------------------------
Total: 00:00:11.4087928
Automating Tests & Publishing Coverage
Now that we have a build script to run our tests and generate coverage results, we can connect our repo to TravisCI and Coveralls. Follow the getting started guides to connect both services to your GitHub account and add your repo. Then, create a .travis.yml
file at the root of your repo with the following contents:
language: csharp
mono: none
dotnet: 2.1
script:
- ./build/travis.sh
I like to wrap my Travis builds in a separate script. This way, if I publish a Docker Container, I can log in to Docker Hub and push the image only on the master branch, which I can detect with environment variables that Travis exposes:
# ./build/travis.sh
#!/bin/bash
set -euo pipefail
SCRIPT_ROOT="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
CAKE_TASK=Coveralls
echo "Building task ${CAKE_TASK}"
"${SCRIPT_ROOT}/../build.sh" -t "${CAKE_TASK}"
We're also going to add a new Cake task to upload coverage results to Coveralls:
Task("Coveralls")
.IsDependentOn("Test")
.Does(() =>
{
if (!TravisCI.IsRunningOnTravisCI)
{
Warning("Not running on travis, cannot publish coverage");
return;
}
MiniCoverReport(new MiniCoverSettings()
.WithCoverallsSettings(c => c.UseTravisDefaults())
.GenerateReport(ReportType.COVERALLS)
);
});
Coveralls has this lovely integration with Travis, where we only need to provide the build ID and it'll figure out the rest of the commit information automatically. If you aren't building in TravisCI, you will need to specify additional information. See the CoverallsSettings
extension methods for details on how to provide this information via Cake.MiniCover
.
Now, we have TravisCI automatically building & testing our code, and Coveralls tracking code coverage!
Conclusion
I hope you found this helpful. I'd love to be able to provide MiniCover support without requiring a tools project, but that requires MiniCover to be split into a core library and a cli tool:
Extract common Library to enable consumption from other tools #31
I want to write a Cake addin for MiniCover like the one that exists for OpenCover. I cannot easily instruct users to add MiniCover as a tool as the way cake downloads tools from Nuget is not compatible with the DotnetCliTool
package type. I could tell users to create a tools project as instructed in the README and pass that to my addin, but I think an easier solution is if we separate the actual instrumentation and reporting logic from the dotnet cli tool frontend. This would allow me to reference it as a dependency in my package and enable users to get simple coverage with something like:
#addin "Cake.MiniCover"
// ...
Task("Coverage")
.IsDependentOn("build")
.Does(() =>
{
MiniCover(tool =>
{
foreach(var project in GetFiles("./test/**/*.csproj"))
{
tool.DotNetCoreTest(project.FullPath, new DotNetCoreTestSettings()
{
// Required to keep instrumentation added by MiniCover
NoBuild = true,
Configuration = configuration
});
}
},
new MiniCoverSettings()
.WithAssembliesMatching("./test/**/*.dll")
.WithSourcesMatching("./src/**/*.cs")
.GenerateReport(ReportType.CONSOLE | ReportType.XML)
);
});
// ...
Then, generating coverage becomes as simple as
./build.sh -t Coverage
# Or on Windows:
./build.ps1 -t Coverage
If you notice any problems with Cake.MiniCover
, definitely open an issue! If you have a feature you'd like to see added, pull requests are welcome as well.
Top comments (1)
For generate tests you can use:
nuget.org/packages/autoCodeCoverage