DEV Community: RENARD Laurent

UI, Composition and Inversion of Control

RENARD Laurent — Thu, 27 Aug 2020 19:50:01 +0000

Designing robust software often involves dividing a complex problem into smaller and flexible bits in order to then compose them into a coherent whole.
In this article we will go through different aspects of software composition thanks to an example built with a web component which renders a greeting message: the famous "hello world" code sample.
Web components specification offers a low level API and does not hide much complexity behind various layers of abstractions as popular UI frameworks can do (this is why you would use such frameworks after all) an therefore make this technology a perfect fit for this tutorial on architectural concepts.
Keep in mind though, that in the context of this article, web components technology is a just a tool to understand the essence of these concepts and prior knowledge of the technology is not mandatory.
Let's start by having a look at the two following functions

const filterEvenNumbers = (numbers) => {
    const output = [];
    for (const number of numbers) {
        if (number % 2 === 0) {
            output.push(number);
        }
    }
    return output;
};

const filterStringsWithE = (strings) => {
    const output = [];
    for (const string of strings) {
        if (string.includes('e')) {
            output.push(string);
        }
    }
    return output;
};

Both work in a similar way yet remain quite different and rely on totally different assumptions: one operates on numbers whereas the other does on strings. They both follow an imperative style you can easily read as a sequence of basic instructions.
Although they do the job, you can quickly see they are not very flexible as they mix together code related to the iterations on theirs data structures and condition checks. It prevents us from sharing any logic between the two functions. However we could quickly let emerge a pattern, especially if we rewrite them as so:

const filterEvenNumbers = (numbers) => {
    const output = [];
    const predicate = (number) => number % 2 === 0;
    for (const number of numbers) {
        if (predicate(number)) {
            output.push(number);
        }
    }
    return output;
};

const filterStringsWithE = (strings) => {
    const output = [];
    const predicate = (string) => string.includes('e');
    for (const string of strings) {
        if (predicate(string)) {
            output.push(string);
        }
    }
    return output;
};

Now we could draw a template into a filter operator:


const filter = (predicate) => (items) => {
    const output = [];
    for (const item of items) {
        if (predicate(item)) {
            output.push(item);
        }
    }
    return output;
};

and write our two functions


const filterEvenNumbers = filter((number) => number % 2 === 0);
const filterStringsWithE = filter((string) => string.includes('e'));

Our predicates become totally independent of the context they are used in while the filter operator does not need to make any assumption on the nature of the data structures it will operate on (beyond the fact they need to implement the iterator protocol). Somehow, we can see the filter operator as a procedure with holes which needs to be filled by the caller.
This principle is often called inversion of control and is at the base of many design patterns such template methods, plugins, dependency injection, etc

UI, data fetching and responsibilities

Let's now consider the following web component:


// component.js
import {createService} from './service.js';

export class Greetings extends HTMLElement {

    static get observedAttributes() {
        return ['name'];
    }

    get name() {
        return this.getAttribute('name');
    }

    set name(val) {
        this.setAttribute('name', val);
    }

    attributeChangedCallback() {
        this._render();
    }

    constructor() {
        super();
        this._fetch = createService();
    }

    async _render() {
        this.textContent = await this._fetch(this.name);
    }
}

For the readers who do not know web components:
The web components specification forces us to declare a component by extending the regular HTMLElement class. We can then define which HTML attributes we want the browser to watch for us thanks to the static getter observedAttributes; and what to do when their values change thanks to the attributeChangedCallback (this is an equivalent for the reactivity/watch mechanism you could find in many UI frameworks) . In our case, we call a custom render function which relies on a data fetch service the component will have created within its constructor.

The service implementation is here a detail but you can imagine something similar to:


// service.js
export const createService = (opts = {}) => async (name) => `Hello ${name || 'Mr. Nobody'}`;

(a basic asynchronous function which takes a string as argument and return a formatted greeting message).
Besides the declarative API (through HTML attributes), we also provide a programmatic API thanks to a property accessor ("name").
We can't however call the constructor ourself (it will throw and error) and must delegate this operation to the browser by registering our custom element into a global registry (this is part of the specification):


// injector.js
export const define = (tag, klass) => customElements.define(tag, klass);

This will allow the browser to create instances of our custom component simply by parsing a tag in the HTML document, or as any regular HTML element, by calling document.createElement(tag).


<!DOCTYPE html>
<html lang="en">
<!-- ... ->
<body>
<app-greetings name="lorenzofox"></app-greetings>
<script type="module">
    import {define} from './injector.js';
    import {Greetings} from './component.js';
    define('app-greetings', Greetings);
</script>
</body>
</html>

You can play around in the following code sandbox by changing the name attribute or with the provided dev tool environment.

Although this example works fine, it is far from being perfect: our component implementation is tightly coupled to a given fetch service. For instance if you wish to test the component in isolation it might be difficult: the service may need to make some network calls, etc. To abstract away the service implementation you would need to hijack the import (with service workers, proxies, etc) to provide a mock or something similar. Jest allows you to do so with global mocks but it is in my opinion an anti pattern and is just a hack which hides a deeper problem in your software.
Writing tests is not an end in itself, but if you encounter some difficulties to test a part of you code, it may be a code smell that your different components are tightly coupled together.
Let's say the requirements have changed and we want to display a different localization message depending on a query string parameter. We have now, various services:


// en.js
export const createService = (opts = {}) => async (name) => `Hello ${name}`;
// fr.js
export const createService = (opts = {}) => async (name) => `Bonjour ${name}`;
// es.js
export const createService = (opts = {}) => async (name) => `Hola ${name}`;
// etc;

The worst that could happen would be a developer in a rush "solving" the problem as so:


export class Greetings extends HTMLElement {
 // ... 
    constructor() {
        super();
        const query = window.location.search;
        const lang = new URLSearchParams(query).get('lang');
        switch (lang) {
            case 'fr':
                this._fetch = createFrService();
                break;
            case 'es':
                this._fetch = createEsService();
                break;
            default:
                this._fetch = createEnService();
        }
    }
// ... 
}

Now our component is coupled to several implementations and to the global object. The constructor carries quite a bit of logic which is almost impossible to test. We could improve somehow the codebase by introducing an indirection point for our services: a single function (createService) which returns the right service based on some parameters. But what if now we want to base the choice of the service on a user setting rather than on the query parameter… yet again, this would require us to change the component code.

Inject the dependency

Ideally we don't want the component (which belongs to some sort of presentation layer) to bear the responsibility of creating/configuring the service which may depends on many parameters out of the component context… and belongs anyway to some sort of business layer.
As we can't call the constructor of a web component and rely on the browser to create an instance of the component this sounds quite challenging, but it is not. First, we can still write our constructor with a default parameter to workaround this issue:


import {createService} from './service.js';

export class Greetings extends HTMLElement {
    //...
    constructor(service = createService()) {
        super();
        this._fetch = service;
    }
    //...
}

This would work as the engine would resolve the passed service as the result of the createService function: we have moved the logic of creating the data fetch service out of the component.
Even better: if we modify slightly the code which registers our component into the global registry we can pass any service:


// injector.js
import {createEnService, createEsService, createFrService} from './service.js';


const resolveService = () => {
    const search = window.location.search;
    const lang = new URLSearchParams(search).get('lang');
    switch (lang) {
        case 'fr':
            return createFrService();
        case 'es':
            return createEsService();
        default:
            return createEnService();
    }
}


export const define = (tag, klass) => {
    const service = resolveService();
    customElements.define(tag, class extends klass{
        constructor() {
            super(service);
        }
    });
};

we have decorated the regular customElements.define function to pass a component which injects the dependency in our component. Now the component is fully independent of any context, so the services are. The only part we need to modify if the requirements ever change is the resolveService function ! This injection of dependencies code is the only one in charge of resolving the appropriate instances in an "omniscient" way.
You can see the whole code here

Testing the component

Instead of relying on global mock hacks we can now easily pass to the component any implementation of the service (including a mock) and test it in full isolation:


import stub from 'sbuts';
import {test} from 'zora';
import {flush, mountComponent} from './utils.js';
import {Greetings} from '../component.js';

test(`when component is mounted, it should not render anything if no name attribute is set`, async t => {
    // given
    const service = stub().resolve(`hello world`);

    // do
    const comp = mountComponent(Greetings, service);
    await flush();

    // expect
    t.eq(comp.textContent, '');
    t.notOk(service.called);
});

test(`when component is mounted, it should render the service message when the name attribute changes`, async t => {
    // given
    const service = stub().resolve(`hello world`);
    const attributeValue = 'lorenzofox';
    const comp = mountComponent(Greetings, service);

    // do
    comp.setAttribute('name', attributeValue)
    await flush();

    // expect
    t.eq(comp.textContent, 'hello world');
    t.eq(service.calls, [[attributeValue]], `service should have been called once with ${attributeValue}`);
});

Fort the record: mountComponent is a test utility function which basically does what the injector in our application does whereas flush is used to make sure any pending Promise is flushed before we make our assertions.
If you wish to see the details you can have a look at the following code sandbox.

Is this a good test ?

Yes… and no. It is a good unit test in the sense it tests the component code in full isolation, abstracting away the service code and making sure whatever is the service implementation, it is called with the right argument. However if for some reason you have to change the interface of a given service implementation


// from
export const createServiceA = (opts) => async (name) => `hello ${name}` 
// to
export const createServiceA = (opts) => async ({name}) => `hello ${name}`;

Your test will keep passing although your application is broken: the test has not caught the regression. But after all, it is not its responsibility to catch changes in a dependency interface as it is meant to test the unit of code related to the web component only.

The point is: when you want loose coupling and introduce dependency injection like patterns you must connect the different parts through interfaces and abstract types.

In Javascript, it is less obvious as the notion of interface is not built-in but if you add a type system (such Typescript) on top of it, your code would not compile and the regression would be caught.
It is then the role of the injector to fix this kind of discrepancies. You can for example use an adapter:


const adapter = (fetch) => (name) => fetch({name});

const resolveService = () => {
    const lang = new URLSearchParams(window.location.search);
    switch (lang) {
        case 'fr':
            // the service with a different interface
            return adapter(createFrService());
        case 'es':
            return createEsService();
        default:
            return createEnService();
    }
};

And Again, there is no need to change either the component code either the service code: the injector connects the dots together!

Conclusion

With this basic example we have seen how a set of architectural patterns can help to create a robust and flexible software without necessarily reaching out to many code branches (if … else … etc): we solve the problem by composition.

There is beauty in simplicity

RENARD Laurent — Tue, 29 Oct 2019 14:59:25 +0000

Last week I finally worked on a test runner for Nodjes based on zora.
I had already written an article inspired by some of the zora's properties and I keep finding interesting how such a small project (in code size) can inspire me new subjects of discussion (I still have few in mind). This one will lead us through some fundamental concepts of Nodejs architecture and general computer programming such event loop, concurrency, parallelism, and how they can be related to the performances of a testing software.

A surprising benchmark

It all started when I added pta to the benchmark in the zora's repository. This benchmark tries to compare speed of execution for various testing frameworks. Performance is clearly at the center of the developer's experience and their productivity when it comes to testing software. Some of the popular frameworks have relative complex architectures involving abstractions such child processes to deliver (not only) top level performances. While zora is at the opposite quite simple but performs much faster according to the aforementioned benchmark.

How can it be ?

The benchmark consists in running N test files, each having M tests. One test would be the corresponding code with the different test runners syntaxes (if I did not make any mistake):

const wait = waitTime => new Promise(resolve => {
  setTimeout(()=>resolve(),waitTime); 
});

test('some test ', async function (assert) {
    await wait(WAIT_TIME); // wait time is a variable of the benchmark
    assert.ok(Math.random() * 100 > ERROR_RATE); // a given percentage of the tests should fail (eg ~3%) 
});

By changing N, M and WAIT_TIME we can mimic what I consider to be the profile of some typical Nodejs applications.

profile small library: N = 5, M = 8, T = 25ms
profile web app: N = 10, M = 8, T = 40ms
profile api: N =12, M = 10, T = 100ms

Each framework runs with its default settings.

Here are the results on my developer machine (MacBook Pro, 2.7GH i5) with node 12 :

	zora-3.1.0	pta-0.1.0	tape-4.11.2	Jest-24.9.0	AvA-2.4.0	Mocha-6.2.1
Library	~100ms	~230ms	~1240ms	~2835ms	~1888ms	~1349ms
Web app	~130ms	~280ms	~3523ms	~4084ms	~2900ms	~3696ms
API	~190ms	~330ms	~12586ms	~7380ms	~3900ms	~12766ms

We can even increase the differences if we use somehow extreme(?) values (N=100, T=10, WAIT_TIME=100ms)

zora	pta	tape	Jest	AvA	Mocha
~450ms	~750ms (1.6x slower)	~104sec (230x slower)	~43.1sec (96x slower)	~24.1sec (53x slower)	~104.5sec (230x slower)

As we will see, the results can actually be predictable, at least for some of the test runners.

The Event Loop and Nodejs's architecture

Nodejs' Javascript engine (like many others) is single threaded and is built around an event loop. There are already many resources online to grasp these two concepts (you can for example refer to the official Nodejs documentation) but to make it short it means:

The main process of a Nodejs program runs within a single thread.
Processing tasks are scheduled with a queue of events. These tasks can be anything like executing a statement, calling the next item of an iterator, resuming a suspended asynchronous function, etc.

The event system is particularly helpful for asynchronous operations as you do not have to block the main thread waiting for a task to complete. You would rather have to launch the asynchronous task and later, when it is over, the scheduler will be notified to enqueue another task: the execution of the callback.

Historically asynchronous tasks were made exclusively through event listeners called, due to their nature, "call me back" or "callback". In modern Nodejs there are newer built in abstractions you can use such async functions and promises or (async)iterators, (async)generator functions, etc. But in essence, the idea is the same: prevent the main thread from being blocked waiting.

Consider the following snippet:

(function fn(){
    console.time('fn timer 1');
    console.time('timer1');
    console.time('timer2');
    setTimeout(() => console.timeEnd('timer1') /* (B) */, 1000); // this won't block the main thread neither the function execution
    setTimeout(() => console.timeEnd('timer2') /* (C) */, 1000); // this won't block the main thread neither the function execution
    console.timeEnd('fn timer') // (A) this will called before the timer is executed
})();

The callbacks will execute after the function fn runs to its completion. The whole program will run in a bit more than 1000ms as the
setTiemout is not blocking: it just schedules on the event loop the execution of the callback function after some elapsed time.

The whole Nodejs architecture is based around these concepts. Let's take the example of a web API.

In a multi threading environment, a request would typically be handled by a thread from its parsing to the sending of the response.
It means once the request has been parsed and the database is processing the query the thread is paused waiting for the database to complete its work, eventually wasting processing resources. Later it is resumed to send the response made of the database result.
It implies you can roughly have as many concurrent requests as threads the server can manage at the same time.

In Nodejs as long as you don't block the event loop the server would be able to handle more requests even within its single thread. It is usually done by using one of the asynchronous patterns to deal with the costly tasks which need access to the disk, the network or any kernel operation. Most of the time, the often called "I/O" operation, is itself delegated to a process which leverage multi threading capabilities like a database server for instance.

Similarly than in our previous example and the setTimeout, the request handler does not have to block the event loop waiting for the database to complete its job, it just needs to pass a callback to execute once the database is done. It means the server can possibly handle a lot of concurrent requests with a single thread, being mostly limited by the database. In a sense, this architecture allows the system to avoid being idle and waste resources.

Concurrency

Concurrency is the ability of a program to start, execute, terminate tasks in an overlapping time. It does not mean the tasks have to run at the same time. It can refer to the ability to interrupt a task and allocate system resources to another task (context switching). Nodejs is a perfect example as you can reach very high concurrency with a single thread.

Now that we are familiar with the callback pattern, let's use async functions and promises instead.

const wait = (time = 1000) => new Promise(resolve => setTimeout(() => resolve(), time));

async function task(label){
    await wait();
    console.log(`task ${label} is done`);
}

The task function may appear to block the main thread but it is not the case. The await statement allows indeed to suspend its execution for a while but it does not prevent the main thread from running another task.

const run = async () => {
    console.time('exec');
    const p1 = task(`task 1`);
    const p2 = task(`task 2`);
    await p1;
    await p2;
    console.timeEnd('exec');
};

// or if it makes more sense

const run = async () => {
    console.time('exec');
    const tasks = [task(`task 1`), task(`task 2`)];
    await Promise.all(tasks);
    console.timeEnd('exec');
};

run();

The last program will run in something close to 1000ms whereas a single task function itself takes 1000ms to run. We were able to execute the two tasks concurrently.

Parallelism

Now let's consider the following function:

// async function is not mandatory here, but it emphases the point.
async function longComputation() {
    console.log(`starts long computation`);
    let sum = 0;
    for (let i = 0; i < 1e9; i++) {
        sum += i;
    }
    console.log(`ends long computation`);
    return sum;
}

This function takes close to 1s to return its result on my machine. But contrary to the task function, longComputation whose code is all synchronous blocks the main thread and the event loop by monopolising the CPU resources given to the thread. If you run the following program

const run = async () => {
    console.time('exec');
    const p1 = longBlockingComputation();
    const p2 = longBlockingComputation();
    await p1;
    await p2;
    console.timeEnd('exec');
};

run();

It will take close to 2s (~1s + ~1s) to complete and the second task won't start before the first one is finished. We were not able to run the two tasks concurrently.

In practice, writing such code is a very bad idea and you would rather delegate this task to another process able to take advantage of parallelism.

Parallelism is the ability to run different tasks literally at the same time. It usually involves running multiple threads with different CPU cores.

Well, actually even with Nodejs you can run multiple threads (or child processes). Let's see an example with the newer Worker Threads API;

worker.js

const {
    parentPort
} = require('worker_threads');

function longComputation() {
    let sum = 0;
    for (let i = 0; i < 1e9; i++) {
        sum += i;
    }
    return sum;
}

parentPort.postMessage(longComputation());

and the main program

const {
    Worker,
} = require('worker_threads');

const longCalculation = () => new Promise ((resolve, reject) => {
    const worker= new Worker('./worker.js');
    worker.on('message',resolve);
    worker.on('error', reject);
});

const run = async () => {
    console.time('exec');
    const p1 = longCalculation();
    const p2 = longCalculation();
    await p1;
    await p2;
    console.timeEnd('exec');
};

run();

Great! This has run in roughly 1000ms. It is also interesting how we have shifted back to the paradigm of the previous section with non blocking functions.

Note: attentive readers will have spotted that the longCalculation creates a new thread worker with each invocation. In practice you would rather use a pool of workers.

How is this related to our testing frameworks ?

As mentioned, speed is a must for the developer experience. Being able to run tests concurrently is therefore very important. On the other hand
it enforces you to write independent tests: if you run tests concurrently you do not want them to mess up some shared data. It is often a good practice but sometimes you need to maintain some state between tests and run various tests serially (one starts when the previous is finished). This can make the design of a testing software API quite challenging...

Let's now try to explain the result we had for our "extreme" case:

Mocha and Tape run test files and tests within a file serially so they will roughly last N * M * WAIT_TIME ~= 100 * 10 * 0.1s ~= 100s (this is consistent)
I can see from the progress in the console that AVA is likely running 4 tests files in parallel on my machine. I think from the documentation that within a file the tests should run concurrently (so that the whole test suite would run roughly in N/4 * WAIT_TIME ~= 25 x 0.1 ~= 2.5s ) but there might be extra cost managing the four child processes (or workers ?) because it is 10 times slower than the expected result.
Jest seems to run 3 test files in parallel on my machine and the tests within a file serially. So I expected N/3 * M * WAIT_TIME ~= 33 * 10 * 0.1 ~= 33s but yet it is slower. Again managing child processes is clearly not free.
Zora and pta run every test concurrently so we can expect the execution time to be related to the slowest test. In practice it takes some time to launch Nodejs, parse the scripts and require the modules. This can explain the little extra time. But the results stay steadily below the second whatever test profile we run.

A small zora

Let's build a small zora to understand how it works (and achieve a high concurrency) and how it tackles the problems mentioned in the introduction of the previous section.

We can write a testFunction function as so:

// test.js
const testFunction = module.exports = (description, specFunction, testList) => {
    let error = null;
    let passing = true;
    const subTestList = [];
    // we return the routine so we can explicitly wait for it to complete (serial tests)
    const subTest = (description, fn) => testFunction(description, fn, subTestList).execRoutine; 

    // eagerly run the test as soon as testFunction is called
    const execRoutine = (async function () {
        try {
            await specFunction({test: subTest});
        } catch (e) {
            passing = false;
            error = e;
        }
    })();

    const testObject = Object.defineProperties({
        // we **report** test result with async iterators... in a non blocking way
        [Symbol.asyncIterator]: async function* () {
            await execRoutine;
            for await (const t of subTestList) {
                yield* t;// report sub test
                passing = passing && t.pass; // mark parent test as failing in case a subtest fails (but don't bubble the error)
            }
            yield this; // report this test
        }
    }, {
        execRoutine: {value: execRoutine},
        error: {
            get() {
                return error;
            }
        },
        description: {
            value: description
        },
        pass: {
            get() {
                return passing;
            }
        }
    });

    // collect the test in the parent's test list
    testList.push(testObject);

    return testObject;
};

and the test harness factory as so

// run.js
const testFunction = require('./test.js');
const reporter = require('./reporter.js');

const createHarness = () => {
    const testList = [];
    const test = (description, spec) => testFunction(description, spec, testList);

    return {
        test,
        async report() {
            for (const t of testList) {
                for await (const a of t) {
                    reporter(a);
                }
            }
        }
    };
};

const defaultTestHarness = createHarness();

// automatically start to report on the next tick of the event loop
process.nextTick(() => defaultTestHarness.report());

module.exports = defaultTestHarness;

The (dummy)reporter being:

// reporter.js
module.exports = testResult => {
    const isFailed = testResult.pass === false;
    console.log(`${!isFailed ? 'ok' : 'no ok'} - ${testResult.description}`);
    if (testResult.error) {
        console.log(testResult.error.stack);
        if (testResult.error.operator) {
            console.log(`operator: ${testResult.error.operator}`);
        }
        if (testResult.error.expected) {
            console.log(`expected: \n ${JSON.stringify(testResult.error.expected, null, 4)}`);
        }
        if (testResult.error.actual) {
            console.log(`actual: \n ${JSON.stringify(testResult.error.actual, null, 4)}`);
        }
    }
};

That's it! You have a whole testing library within less than 100 lines of source code which can use whatever assertion library as long as it throws an error (the assert module from Nodejs' core is a good candidate !).

It will report failures: "where?", "what?" and "why?"

const assert = require('assert').strict;
const {test} = require('./run.js');

test(`some test`, () => {
    assert.deepEqual([1, 2, 3], [1, 2, 4], `array should be equivalent`);
});

will output:

It will run every test concurrently and will likely be faster than all the other mega bytes sized test runners

test(`some async test that shows concurrency`, async t => {

    let foo = 'bar';

    t.test(`nested async`, async t => {
        await wait(100);
        assert.equal(foo, 'baz', 'see changed value although started before');
        foo = 'whatever'
    });

    t.test(`change foo faster`, t=>{
        assert.equal(foo, 'bar');
        foo = 'baz';
    })

});

Yet it will allow you to control the concurrency of you test with regular javascript control flows

test(`some serial test`, async t => {
    let foo = 'bar';

    // we specifically wait for that test to complete with the "await" keyword ...
    await t.test('nested inside', async t => {
        await wait(100);
        assert.equal(foo, 'bar', 'see the initial value of foo');
        foo = 'whatever';
    });

    // to start this one
    t.test('run only once "nested inside" has finished', () => {
        assert.equal(foo, 'whatever', 'see the changed value');
    });

});

If you wish to play with this basic test runner, you can fork the following gist and run the test program with node: node test_program.js

Conclusion

We have reviewed Nodejs' architecture and saw how it can allow high concurrency without necessarily involving parallelism. We have placed it in the context of a testing software and saw how we could give a high quality user experience to the developer and greatly improve their productivity.

We can also discuss whether parallelism has an added value in the context of Nodejs testing experience. We already saw that it may not be the case regarding the performances. Of course you could find some use cases where parallelism could bring you better performances. Or you could argue the test function in the benchmark is not "blocking enough" to be realistic (you would be right!) but as we said earlier, if you need parallelism in your tests because the code you are testing is slow, you are probably doing it wrong.

In practice I have personally been using zora (or pta) for a wide range of use cases and never had any performance issue:

In ship-hold, we run a whole range of integration tests against a database server below a second.
In mapboxgl-webcomponent, we run browser automation (screen shots capture, etc) within few seconds (this might actually be considered slow).
In smart-table, we run many unit tests in a second.
pta is tested by itself and the test suite contains child processes to run pta's CLI as a binary, all this in less than 2 seconds.

On the other hand, child processes have other interesting properties from a testing perspective, naming isolation. It allows you to run a given set of tests in an isolated, sand boxed environment.
However, it also leaves you with few new issues to address (stream synchronisation, exit codes, etc) making the code base inevitably grow. I would not say AVA is minimal(14.8mb), neither is Jest(32mb). Of course they offer way more "features" than our few bytes test runner. But are "runs previously failed tests first" or "re-organizes runs based on how long test files take" really required when a whole test suite runs within a pair of second.

The title refers to our ability, as developers, to sometimes over engineer solutions where simplicity is just what we need.

Tools and the design of a testing experience

RENARD Laurent — Wed, 04 Sep 2019 14:39:10 +0000

Tools are everywhere in a developer's workflows. NPM, the JavaScript package manager is indeed full of productivity tools which aim at improving software quality and developers' efficiency. However it is not the only source as we will see later in this article.

Some tools may directly improve user facing products such a code minifier (terser) which helps to reduce the amount of code shipped within a production web application. But most of the time a tool rather helps the developers by improving their own experience and by making the code easier to maintain, debug and ship. In that sense you can say it also helps the end user indirectly. A bundler (rollup) or packager (webpack) for instance, will allow developers to split a code base into smaller chunks easier to reason about.

The scope of a tool may vary as well. Some are very focused on a particular problem (mkdirp) while others try to build a whole user experience around a wide range of problems (create-react-app).
In the second case, we may not realize but the tool becomes really the expression of opinions or processes on how to solve a set of problems. Therefore such a tool usually has to make some trade-off and may harm the user experience depending on the requirements.

In this article, instead of using an existing testing framework full of features, we are going to tailor our own testing experience based on actual problems and requirements while they arise during the development cycle of a software.

Tailoring a test experience

I have chosen the testing experience as leitmotiv because it is quite a challenge as it may involve many different topics (code transformation, reporting, different running environments, performances, etc) and may vary a lot between two different use cases. It is probably the reason you have already many testing frameworks in the JavaScript ecosystem.

Monolithic design vs The UNIX philosophy

Popular JavaScript testing frameworks come usually with a lot of features. As stated earlier these features are somehow opinions on what problems you may encounter and how to fix them so you don't have to think about it and can focus on your tests. They usually provide configuration settings and programmatic extension points so you can tweak your testing experience based on your needs and bring some flexibility to the worflow.

On the other hand, they may not be flexible enough or introduce extra complexity if your needs are a little bit out off the frame.
For example AvA automatically transpiles ESM syntax on your test files. It may be very useful if you write your tests in some way (you don't have to configure anything to get the transpilation done!) but it may be difficult to bypass or confusing to come out with a set up if you write your tests in another way. That is an example on how an opinion may go against flexibility.

Another approach, is the UNIX philosophy which

favors composability as opposed to monolithic design

The idea is to compose small focused programs together to achieve a greater goal.

Compared to our AvA's example you can build a testing experience with three components as so.

transpiler -> test runner -> reporter

And if you don't need the transpiler, you can just remove it from the pipeline.

This is very flexible as long as every component is designed to use a common interface (text streams).

A simple Node program

The boss comes to you and asks:

I want you to write a functional Node Math library.

You agree on the first stretch to implement an add function to perform the sum of two numbers and which would support partial application. You come with the following implementation (The implementation is actually a detail here).

//src/index.js

module.exports = (a, b) => {
    if(b === void 0){
        return x => a + x;
    }

    return a + b;
};

you write the following testing program.

//test/index.js
const add = require('../src/index.js');
const {test} = require('zora');

test(`add(a,b) - should sum the two arguments provided`, t => {
    t.eq(add(2,4), 6, `2 + 4 = 6`);
});

test(`add(a) - should support partial application`, t => {
    const add_two = add(2);
    t.eq(add_two(3), 6, `2 + 4 = 6`); // let's make that one fail
    t.eq(add_two(11), 13, `2 + 11 = 13`);
});

You will have noticed for the testing program we are using zora. Following the UNIX philosophy, it is a small library I wrote which is dedicated on writing JavaScript testing programs: not running them, not transforming source code, not printing colours in the console, etc. Of course it comes with its own opinions but will be particularly useful for this short essay as it is very focused on one single problem compared to other full featured frameworks.

You can run the test program with Node thanks to the command:

node ./test/index.js

You will see the following output in the console

TAP version 13
# add(a,b) - should sum the two arguments provided
ok 1 - 2 + 4 = 6
# add(a) - should support partial application
not ok 2 - 2 + 4 = 6
  ---
    actual: 5
    expected: 6
    operator: "equal"
    at: " Object.<anonymous> (/Volumes/data/article-playground/test/index.js:8:1)"
  ...
ok 3 - 2 + 11 = 13
1..3

# not ok
# success: 2
# skipped: 0
# failure: 1

The output is a text format called TAP (Test Anything Protocol). It gives you a status on each test of your program and in case of a failure, you'll have the location of the failure and the reason it failed so you can fix your test/source code. After all it is all what you can expect from a testing program.

Composing with a pipeline

Arguably, the output is not very human friendly (no colour, the passing tests may be considered as noise, etc). Most of the testing frameworks come with a set of reporters you can choose from depending on your preferences. In UNIX philosophy you will ask another program to process this output stream. TAP is a widely spread text protocol and not only in the JavaScript community so that you should find plenty of tools able to parse and process a TAP stream.
For example you can install tap-summary from NPM registry and now type the command:

node ./test/index.js | tap-summary

You will get the following output

If you need something different there is no problem. Just search for tap reporters in NPM or install a binary coming from a different technology. That's the beauty of delegating the reporting task to a different process.

Exit codes

Zora is itself platform agnostic: it is not in charge of running your testing programs. You should be able to do it with any JavaScript run time which supports the Ecmascript specification (edition 2018): Node >= 10, modern browsers, etc. However in a Node environment, one can expect the process executing the testing program to exit with a code different than 0 in case of a failure in the tests. That is actually a requirement in many continuous integration platforms to mark a build as failed and avoid false positives.
However if you print the exit code of your testing program you will get 0.

node ./test/index.js; echo $?; 
# > 0

Thankfully by delegating the reporting part to a different process more "platform aware" we can remedy to this eventual problem as the exit code will be the one returned by the last process in the pipe:

node ./test/index.js | tap-summary; echo $?; 
# > 1

More advanced program

The following week, you agree to deliver the multiplication operator. As your code base grows, your team decide to split the code into different files to better organize the code and ease the maintenance. You have now the following implementation.

// src/util.js
exports.curry = fn => (a, b) => b === void 0 ? x => fn(a, x) : fn(a, b);

// src/addition.js
const {curry} = require('./util');

module.exports = curry((a, b) => a + b);

// src/multiplication.js
const {curry} = require('./util');

module.exports = curry((a, b) => a * b);

// src/index.js (the entry point of the library)
exports.add = require('./addition');
exports.multiply = require('./multiplication');

And the testing part of the project will also reflect the new organization.

// ./test/addition.spec.js
const {add} = require('../src/index.js');
const {test} = require('zora');

test(`add(a,b) - should sum the two arguments provided`, t => {
    t.eq(add(2, 4), 6, `2 + 4 = 6`);
});

test(`add(a) - should support partial application`, t => {
    const add_two = add(2);
    t.eq(add_two(3), 6, `2 + 4 = 6`); // let's make that one fails
    t.eq(add_two(11), 13, `2 + 11 = 13`);
});

and

// test/multiplication.spec.js
const {multiply} = require('../src/index.js');
const {test} = require('zora');

test(`multiply(a,b) - should multiply the two arguments provided`, t => {
    t.eq(multiply(3, 4), 12, `3 * 4 = 12`);
});

test(`multiply(a) - should support partial application`, t => {
    const time_three = multiply(3);
    t.eq(time_three(4), 12, `3 * 4 = 12`);
    t.eq(time_three(10), 30, `3 * 10 = 30`);
});

Neat! A new problem arise though. If we keep using Node as the runner we now need to run several testing programs (one for every *.spec.js file). A naive approach would be to simply run every file:

node ./test/multiplication.spec.js && node ./test/addition.spec.js

However this solution is not very efficient and we probably want to consider all our tests as a whole.

The simple solution

We can create an entry point for our testing program exactly the same way we already do for our library

// ./test/index.js
require('./addition.spec.js');
require('./multiplication.spec.js');

And that's it, we can now run all the tests with a single command and still pipe the output to another process.

node ./test/index.js | tap-summary

Another good point is that many tools which perform code transformation require a single entry point. So if we need extra build step on our testing program we are all good.

We can also decide to run a single test file which usually gathers functionally similar tests together. In the same way we can decide to comment out some files very easily.

The small downside however, is that we have to maintain this entry point: for example, we must not forget to add the require statement after we have added a new test file.

The funny solution

The previous solution showed us that all what we need is a program to dynamically require files. Interestingly, tape another popular testing library (and which has a lot in common with zora - zora was inspired by tape) comes with a command line interface which basically does what we need. So if we install tape we can basically use its test runner as so:

tape ./test/*.spec.js

Both libraries are very small according to package phobia (tape metrics and zora metrics) yet it probably does not make sense to have both installed.

The scripting solution

Interpreted languages with access to system APIs such JavaScript or Python are very powerful automation tools: they offer a lot of already built packages (thanks to NPM in our case). Moreover, once you are
used to the core modules (fs, path, etc), you can quickly generate custom tools and command line interfaces.

However the operating system itself (at least in UNIX systems) comes with a rich set of scripting capabilities through the shell, the default Bash interpreter and its builtins. I am currently learning more thoroughly Bash as it offers more possibilities for short scripts in the long term. Node is not necessarily available everywhere whereas you can use Bash in CI scripts, on remote servers, whit Docker images and you already use your terminal anyway, at least to run simple commands.

So in this section we are going to create in Bash that file which will require dynamically the spec files to showcase few of the possibilities Bash can offer.

Consider the following file (./scripts/tester.sh)

#!/usr/bin/env bash

# fail on first failing pipeline
set -e;

# set the debug file path in the project based on an environment variable (or use the default)
debug_file=${DEBUG_FILE:-$PWD/test-debug.js}

# clean existing debug file if any
rm -f $debug_file;

# use passed arguments for spec files definition or use default glob ./test/*.spec.js
spec_files=${@:-$PWD/test/*.spec.js};

# generate the debug file depending on the input parameter
for f in $spec_files;
  do echo "require('$f');" >> $debug_file;
done

# run the debug file with node
node $debug_file;

you can make it executable thanks to the command

chmod +x ./scripts/tester.sh

and run it

./test/tester.sh

They are different ways to make the latest script more user friendly in your daily workflow and more portable. You can for example create an alias for the current session

alias t="./scripts/tester.sh"

So now you can run your tests by simply typing t in your terminal.

The script itself is more or less self explanatory: it creates a fresh debug file (test-debug.js) which will require spec files based on the argument. If no argument is provided it will require all the
files matching the pattern ./test/*.spec.js. Finally, it will run the debug file with Node.

You can overwrite the debug file name thanks to an environment variable and you can require a subset of the spec files by passing a list of arguments to the script.

export DEBUG_FILE="test.js";
t ./test/{addition,multiplication}.spec.js

If you want a minimalist reporter to only print in the console the failing tests with their diagnostic, you can pipe the output into a grep command

t | grep '^not ok\|^\s'

will output

not ok 2 - 2 + 4 = 6
  ---
    actual: 5
    expected: 6
    operator: "equal"
    at: " Object.<anonymous> (/Volumes/data/article-playground/test/addition.spec.js:8:1)"
  ...

The smart solution

It is less known but when you call the Node executable you can pass some options. One particularly handy for us is the require option which allows to load some modules before the actual script runs. And it supports glob patterns! So if you type the following command:

echo "process.exit(0);" | node -r ./test/*.spec.js

It is a little bit like you would run the following Node program

require('./test/addition.spec.js');
require('./test/multiplication.spec.js');
// and other *.spec.js files if any
process.exit(0);

It will basically run all the spec files and exit the process with the status code 0 if the program managed to run to its completion. You can of course change the pattern if you want to run a subset of the test files.
And if you want a different exit code in case of a test failure, again, just pipe the output to a specialized reporting process.

echo "process.exit(0);" | node -r ./test/*.spec.js | tap-summary

The icing on the cake: code coverage

It is sometimes useful to know which part of your source code is tested and more importantly which one is not. There are various libraries in the JavaScript world able to do so. Some require code instrumentation: a process which transforms your code to add "counters" around every line in order to know how many times a line is traversed.nyc (and Istanbul) are the most famous. As these libraries require an initial build step they may add a layer of complexity in the process.

Lately, V8 (Chrome's JavaScript engine which is shipped within Node) has been bundled with code coverage capabilities. Thanks to c8 module you can somehow rely on this "native" feature of the engine to measure you code coverage.

echo "process.exit(0);" | c8 node -r ./test/*.spec.js | tap-summary

Conclusion

With a simple command line, by composing different small and focused software together we have managed to build our own flexible testing experience.

It includes everything we need and nothing more: an assertion library (zora), a free and flexible test runner (Nodejs), code coverage (c8), custom reporting (tap-summary) and exit code handling (tap-summary) while it keeps our dependency tree exactly to what we can expect.
Moreover, if at anytime we want to change a component or simply remove it, it is straightforward and does not rely on any complex configuration file. In the same way you can add other components when the need appears (babel, typescript, etc).

In the next episode we are going to see how it goes in the browser...