DEV Community: linkurious-dev

Choosing the right tools to test a visualization library

linkurious-dev — Wed, 24 Jun 2020 10:58:06 +0000

Stories from the trenches

At Linkurious, we’ve designed Linkurious Enterprise, a platform that leverages the power of graphs and graph visualizations to help analysts and investigators around the globe fight financial crime.

One of the main features of Linkurious Enterprise is a user-friendly graph visualization interface aimed at non-technical users.
In 2015, unhappy with the state of JavaScript graph visualization libraries, we started developing our own: Ogma.

Ogma is a Javascript library we built that is focused on network visualization: you may have seen networks visualized before in Javascript with other tools like D3.js or Sigma.js , but for us it was very important to enable some specific feature that were not available yet in other libraries, hence the creation of the Ogma visualization library from the ground up.

The problem

As part of our journey while developing our graph visualization library, we encountered many challenges. One of these challenges is: what is the best way to test a visualization library?

Testing visualization libraries is important because the complexity of the codebase is very high: several algorithms, multiple renderers and a vast API surface makes it very hard to keep things simple enough to be managed without an automated system.
Given this complexity, it is non-trivial to come up with a testing solution for all the many aspects of the visualization library, given the amount of things to test and the available resources for it.

Our solution

We think that testing a library is not just about testing the library itself, but testing the whole experience of using it as a developer:

coding with the library itself
reading the documentation
using the working examples

After several iterations, we ended up with a mix of approaches that we think works great:

unit tests
integration tests
rendering tests
examples testing

Unit tests

How many times have you been told to write unit tests? So it’s not a surprise to hear that we’re using them I guess.

The interesting thing is that we are not 100% focused on these: writing unit tests is pretty expensive as it requires to test in isolation each single bit of the library in multiple scenarios, with great use of time and resources.

Because of that, we’re using a more pragmatic approach instead as Guillermo Rauch puts it:

Guillermo Rauch

@rauchg

Write tests. Not too many. Mostly integration.

16:43 PM - 10 Dec 2016

185 513

Integration tests

Integration tests are not so different from unit tests, some people even consider them unit tests as well. The biggest difference from unit tests is that Integration tests are run against the external API of the library rather than to specific modules.

This approach leads to testing a wider spectre of code and it is the closest thing from a library point of view, to check and keep control of bugs from the developer experience.

Developers using the library are going to see only the API as a gateway to the state and behaviour for their goal: that is why we want to really stress as much as we can this side to catch both new bugs and regressions before the releasing process.

For this reason the Ogma API is covered as much as possible and the coverage tool here is fundamental to see if all paths are reached before going into the rabbit hole of inner modules - where unit tests are kicking in for specific and edge case testing.

Mind cross-browser compatibility

While Ogma can run also in a Node.js process, the fact that it is a visualization library makes it extra important that it works cross-browser flawlessly. That’s why all integration tests are run against a wide set of browsers and operating systems: from Internet Explorer 11 on Windows 7 to the latest Chrome version on MacOS.

Tools:

mocha.js (in combination with chai.js) for both NodeJS and cross-browser environments
Browserstack (or any alternative is good as well)
nyc for code coverage tool

Process:

Also these tests are run for every commit on PRs on each repository within the CI for both NodeJS and cross-browser environments.

A pre-push hook is in place to run it locally in NodeJS to prevent developers to break things while pushing it upstream.
The .only check is also in place here.

Rendering tests

Once we check that all code is good it should be enough to release right? Well, not yet.

The most important feature of a visualization library is its… rendering output.

We can test rendering instructions sent to the rendering engine and validate that, but the final result is something that can not simply be spot from logs or code.

That’s why rendering tests becomes important to support the QA of the library and reduce the amount of regression bugs on the library. Ogma provides three different rendering engines (SVG, Canvas and WebGL) and each browser has its own quirks for each that we need to spot before releasing a new version.

In this context tools like puppeteer or selenium-like comes very handy to quickly put together visual regression tools: a test is a Web page with a network visualization with specific attributes which gets rendered and exported as image, then diff’d with some reference images.

Tools:

puppeteer or Browserstack
pngjs and pixelmatch for image diff

Process:

These tests are run at every commit on PRs on the CI.
Because rendering engines may differ, tests and reference images are different for each platform.

Documentation examples testing

Here’s at the end of our post and covering the most undervalued side of testing: resources given to the developer, other than the library!

Documentation is hard to test automatically, there are probably tools out there that are smart enough or leveraging some sort of NLP to verify documentation text, but in our case it’s too complex to handle it right now: a human reading and checking is still the best we can do for text, we can actually check the types and definitions, combining this with Typescript definition file.

Often for the TS definition file, it is required to expose only those types necessary to interact with the API, so some internal types are stripped out of the definitions and an integrity check is performed on it to verify the consistency.

Another side of the documentation are examples, often used by developers to better understand the features of the library: it is very important that these examples don’t break and that developers are able to run them locally.

Tools:

puppeteer for examples checking (primitively checks for exceptions thrown)
Typescript + JSDoc to generate the right signature types
tsc for definition file integrity check

Process:

Examples tests are run at every commit on PRs on the CI.
Definition files are checked both on pre-commit hook and on the CI at each commit.
The codebase has been recently ported from Javascript to Typescript so signature check is permanent during development - probably a theme for another blog post in the future ;)

Conclusions

The different approaches of each type of testing helped to span across the whole developer experience, from the documentation to the effective library rendering results providing an effective tool to detect and handle breaking changes before they could reach the release script.

Creating the right mix of tests has been a long effort of the team during the past years, which ended up consolidating recently with pre-hooks and a re-designed CI flow to provide quick feedback.

The mix of different levels of testing helped us tame the complexity of the library, reducing the number of bugs that we receive, preventing the number of regressions and increasing the speed for enhancements, with the resources available in the company.

We hope you’ve enjoyed the blog post and got some value out of it. If you have any question, suggestion or comment, please let us know. And remember we’re always looking for nice people who like to test, so let us know if you like to write code and test it!

To WASM or not to WASM?

linkurious-dev — Fri, 12 Jun 2020 08:06:34 +0000

A WASM benchmark story

At Linkurious, we build Linkurious Enterprise, a Web platform that leverages the power of graphs and graph visualizations to help companies and governments around the globe fight financial crime.

One of the main features of Linkurious Enterprise is a user-friendly graph visualization interface aimed at non-technical users.

In 2015, unhappy with the state of JavaScript graph visualization libraries, we started developing our own: Ogma.

Ogma is a JavaScript library we built that is focused on network visualization, providing excellent rendering and computing performances. You may have seen networks visualized before in JavaScript with other tools like D3.js or Sigma.js, but for us it was very important to enable some specific feature and improve specific performance metrics not available on other libraries, hence the creation of the Ogma visualization library from the ground up.

The problem

Ogma has been designed to work with state-of-the-art algorithms to provide the best performance in the field of network visualizations, from a first-class WebGL rendering engine, to the adoption of WebWorkers to improve the interactivity of the library on long running tasks and finally with top-class graph layout algorithms implementations.

Since the first announcement, WebAssembly promised great performances – comparable to native ones – with very little effort from the developer himself other than developing the source code into a native performance language to get the best results on the Web.
After some time and many more announcements on the WebAssembly side, we decided to give it a try and run a thorough benchmark before jumping on the (performant) WASM bandwagon.

The perfect candidate for this kind of investigation are graph layouts: they are CPU intensive, crunching numbers over and over until a solution converges from it.
The promise of WASM is exactly to solve this kind of problem with better memory and CPU efficiency at a lower level compared to the JavaScript interpreter.

Our investigation

Our investigation focused first on finding a candidate to benchmark a typical graph layout algorithm, which can be easily ported to different languages using similar structures.
The choice landed on the n-body algorithm: this algorithm is often the baseline of many force-directed layout algorithms and the most expensive part in the layout pipeline. Solving this specific part of the pipeline would provide great value at the overall force-directed algorithms Ogma implements.

The benchmark

As Max De Marzi said on his blog last summer in 2019:

There are lies, damned lies, and benchmarks.

Building a fair benchmark is often not possible because it’s hard to reproduce real world scenarios: creating the right environment for a complex system to perform is always incredible hard because it’s easy to control external factors in a laboratory benchmarking, while in the real life many things concur to the final “perceived” performance.

In our case our benchmark will focus on a single, well defined, task: the n-body algorithm.
It is a clear and well-known defined algorithm used to benchmark languages by reputable organizations.

As any fair benchmark comparison, there are some rules we defined for the different languages:

The code structure should be similar for the different implementations
No multi-process, multi-thread concurrency allowed.
No SIMD allowed
Only stable versions of the compilers. No nightly, beta, alpha, pre-alpha versions allowed.
Use only the latest versions of the compilers for each source language.

Once defined the rules, it is possible to move to the algorithm implementation. But first, it’s necessary to decide which other languages will be used for the benchmark:

The JS competitors

WASM is a compiled language, even if declared as “human readable” assembly code, it’s not a (mentally) sane choice for us to write plain WASM code. Therefore we ran a survey for the benchmark and we picked the following candidates:

The n-body algorithm has been implemented in the 3 languages above and tested against the JavaScript baseline implementation.

On each implementation, we kept the number of points at 1000 and ran the algorithm with different numbers of iterations. For each run, we measured how long it took to perform the computations.

The setup of the benchmark was the following:

NodeJS v. 12.9.1
Chrome Version 79.0.3945.130 (Official Build) (64-bit)
clang version 10.0.0 - C language version
emcc 1.39.6 - Emscripten gcc/clang-like replacement + linker
cargo 1.40.0
wasm-pack 0.8.1
AssemblyScript v. 0.9.0
MacOS 10.15.2
Macbook Pro 2017 Retina
Intel Dual Core i5 2,3 GHz, 8GB DDR3 with 256GB SSD

Not top of the class machine for a benchmark, but we’re testing a WASM build which is going to be run in a browser context, which usually does not have access to all the cores and RAM memory anyway.

To put some spice on the benchmark, we produced several versions of each implementation: a version where each Point in the n-body system has a 64 bit numeric coordinate representation, and another version with a 32 bit representation.

Another note to consider is probably the “double” Rust implementation: originally in the benchmark a “raw” Rust “unsafe” implementation was written without using any particular toolchain for WASM. Later on, an additional “safe” Rust implementation was developed to leverage the “wasm-pack” toolchain, which promised easier JS integration and better memory management in WASM.

Crunching the numbers

To crunch the numbers, 2 main environments have been tested: Node.js and a browser environment (Chrome).
Both benchmarks run in a “warm” scenario: the Garbage Collector has not been reset before each benchmark suite. From our experiments running the GC after each suite had no particular effects on the numbers.

The AssemblyScript source was used to build the following artifact:

The JS baseline implementation
The AssemblyScript WASM module
The AssemblyScript asm.js module¹

Crunching the numbers in Node.js shows the following scenario:

And then run the same suite in the browser:

The first thing we noted was how the AssemblyScript “asm.js” performs slower than other builds. This chart did not make it clear enough how well or bad other languages were doing compared to the JS implementation, so we created the following charts to clarify:

There is a distinction here between 32 and 64 bit, which may lead to the idea that JS numbers can have both representation: numbers in JS - our baseline - are always at 64 bits, but for the compilers to WASM it may make some difference.

In particular, it makes a huge difference for the AssemblyScript asm.js build at 32 bit. The 32 bits build has a big performance drop compared to the JS baseline, and compared to the 64 bit build.

It is hard to see how the other languages are performing compared to JS, as AssemblyScript is dominating the chart, therefore an extract of the charts was created without AssemblyScript:

The different numeric representation seems to affect other languages too, but with different outcomes: C becomes slower when using 32 bit (float) numbers compared to the 64 bits (double), while Rust becomes consistently faster with 32 bit (f32) numbers than with 64 bit (f64) alternative.

Poisoned implementations?

At this point one, question may come to mind: since all tested WASM builds are quite close to the JS implemented code, would it be possible that the native implementations are slower themselves and the WASM builds just mirror that?

Native versions of the implementations were always faster than its JS counterpart.

What has been observed is that the WASM builds perform slower than their native counterpart, from a 20% up to a 50% performance penalty - performed on a reduced benchmark version with 1000 iterations:

In the measurements above, the native measures are counting also the bootstrap time, while on the WASM measurement that time has been taken out.

Conclusion

The performance gain we measured with Rust (both implementations) is up to 20% compared to the baseline JavaScript implementation - measured on average.
This may sound like a win for Rust, but is actually a very small gain compared to the efforts required.
What did we learn from that? We concluded that writing JavaScript code carefully leads to high performance without the need to jump to new languages.

Learning new languages is always a good thing, but it should be for the right reason: performances are many times the “wrong” reason, as they are more affected by whole design decisions rather than compiler or micro-benchmark optimizations.
As a field experience, we did change language from JavaScript to TypeScript to write our own force-layout algorithm: what we improved was the quality of the codebase, rather than performances, which we measured during the porting and brought a marginal 5% gain, probably due a refactoring of the algorithm - we’ll cover that in a future blog post.

If you’re interested in performances, and JavaScript, you may also find this talk from the DotJS 2019 conference quite interesting, bringing similar results to ours.

Footnotes

1: Interesting to note how the “AssemblyScript asm.js module” was not actually fully asm.js compliant. We tried to add the “use asm” comment on top of the module, but the browser refused the optimization. Later on, we discovered how the binaryen compiler we used does not actually target full asm.js compliance, but rather some sort of efficient JS version of WASM. ↑