Let me present you today a whole new code coverage tool, aiming at accuracy, simplicity, flexibility and speed, named One Double Zero.
Why another code coverage utility?
Fair enough: both nyc and c8 are widely used tools, so why would we need another code coverage utility?
The short answer is: because none of them is satisfying.
For the long answer, let's start by establishing what a good code coverate tool should be, and, then, let's see how nyc, c8 and One Double Zero compare to this definition.
- A code coverage tool should be accurate: it should report relevant - and only relevant - uncovered branches, functions and statements, accurately.
- A code coverage tool should be simple: it should be straightforward to use, to learn and to teach.
- A code coverage tool should be flexible: it should be usable with any kind of project structure, and regardless of how test and tested binaries are built and executed.
- A code coverage tool should be fast: it should not get in the way of Test Driven Development and should allow developers to test coverage often.
nyc
nyc is accurate
Some minor inconsistencies - like named export statements being sometimes counted as statements, sometimes ignored- prevent nyc from being perfectly accurate, though.
nyc also totally ignores files that consist of only non relevant statements, like files that only export TypeScript types, instead of emitting them as empty files. This is a design choice that actually makes it appropriate to cover TypeScript projects.
nyc is not simple
nyc is actually quite confusing to use. It exposes multiple options that seem to overlap with each other (all, exclude, excludeNodeModules, extension, include) alongside a cryptic undocumented one (excludeAfterRemap) that must sometimes be set to true, and sometimes be set to false, depending on unclear circumstances. This complexity motivated a lot of developers to just share and reuse curated configurations, that they don't understand, and get away with it.
nyc is flexible
nyc supports any kind of project structure and test suite execution strategy…as long as one can overcome its confusing usage.
nyc is not fast
This is to be expected, and it does not say anything bad about the intrisic quality of nyc: nyc was written at a time where V8 was not able to provide any execution coverage data, which means that nyc has to instrument the included scripts before executing the passed command.
This instrumentation step is responsible for most of the overhead.
c8
c8 is not accurate
c8 regularly misses some true positive, the positions of the issues are not always correct and it doesn't support multiple statements per line. It also considers that every single file contains at least one function, and makes no distinction between the parts of the code that are relevant at runtime, and those that are not: empty lines, comments, expressions and type declarations, among others unrelevant items, are all considered as relevant statements - regardless of them being statements or not.
One of the direct consequence of this behaviour is that a file exporting only TypeScript types is marked by c8 as uncovered, instead of non-relevant, making the coverage metric drop. As such, it is, barely usable with TypeScript projects.
c8 is not simple.
c8 borrows most of the design philosophy of nyc, and pursues on the confusing usage. It exposes the same options as nyc, and adds even more overlapping ones to the mix: src, and the official but non-documented allowExternal. Using it is even more complex than using nyc and the consequences are exactly the same : developers ended up sharing and reusing curated configurations, that they don't understand.
c8 is flexible, until it is not.
c8 supports any kind of project structure and test suite execution strategy…as long as one can overcome its confusing usage.
However, it fails as soon as it is provided a configuration file and is executed from a subdirectory, which makes it inappropriate to execute discrete tests locally.
$ cd src/test/unit/cases
$ c8 ts-node a-specific-test-case.ts
=============================== Coverage summary ===============================
Statements : Unknown% ( 0/0 )
Branches : Unknown% ( 0/0 )
Functions : Unknown% ( 0/0 )
Lines : Unknown% ( 0/0 )
================================================================================
Nota that this may be a bug that may eventually be fixed (see https://github.com/bcoe/c8/issues/498 and https://github.com/bcoe/c8/issues/527).
c8 is fast.
It benefits from V8 execution coverage data, which means that it does not have to instrument anything, and doesn't do any parsing.
So, what about One Double Zero
One Double Zero is accurate
One Double Zero operates on the AST of the source files, instead of operating on the collected coverage data:
- first, it executes the passed command, collecting coverage data along the way;
- then, it parses the source files;
- finally, it challenges every relevant AST node against the collected coverage data.
This strategy allows One Double Zero to organically support multiple statements per line, to understand the difference between statements and expressions and, more generally, to be as accurate as the AST is.
Incidentally, files that only consist of exported types are organically considered by One Double Zero as empty and are emitted as such in the coverage summary and report: they are visible, but they have no impact on the metrics.
One Double Zero is simple
One of the most spectacular things about One Double Zero is its simplicity: it only needs to know what the source files are; everything else is deduced from the AST of the source files, and from the source maps of the executed scripts if they exist.
Additionally, its rational can be explained very easily:
Coverage consists at executing a command and, then, check how many times each branch, function, line and statement of a given arbitrary set of source files was hit during the execution of the command.
Hence, a code coverage tool only needs to be provided an arbitrary set of source files, and a command to execute.
Which, in One Double Zero command semantic, translates organically to:
$ odz --sources=src/**/*.ts npm t
Obviously, One Double Zero also supports reporting related options (like thresholds, output directories or reporters), but they are just that: options.
One double zero is flexible
By making no other asumption than the only things required by a test coverage tool are a command to execute and a list of source files to check coverage for, One Double Zero supports any kind of project structure and binary building and any execution strategy that one can imagine.
It also supports sub-directory execution, CLI flags, and configuration files written in TOML and JSON.
One Double Zero is fast
One Double Zero is about as fast as c8. Like c8, it benefits from V8 execution coverage data, which means that it does not have to instrument anything. It spends most of its time parsing the source files, which it compensates by computing the actual coverage only for the relevant AST nodes.
Should I move to One Double Zero?
Definitely. It is arguably better than both nyc and c8, comes with an extensive documentation, a JavaScript API, a state-of-the-art project, and including it in a toolchain is a matter of minute - it took ten minutes to replace nyc with One Double Zero in the Twing project; note a fascinating fact here: One Double Zero was able to detect that one of the file of the Twing project was unused at all, which allowed us to remove it, something that nyc failed at detecting.
One Double Zero can be downloaded through npm.
Thanks for reading, happy coding, happy covering!
Top comments (0)