Property Based Testing Framework for Node

The Pragmatic Programmer introduces a method of testing called property-based testing, in which an example is given in Python, using the framework hypothesis.

The usage of hypothesis is very intuitive and simple, and presents the concept of property-based testing perfectly. So I also wanted to find an equivalent alternative in Node. Two of them have high star ratings on Github, JSVerify with 1.6K stars and fast-check with 2.8K stars. So I took some time to study fast-check a little bit and try to get closer to my daily work.

This article is a recap, and a simple example to document the experience.

Why Property-Based Testing?

Before providing examples, let's explain why we use property-based tests. In fact, I don't like the term property-based. In my words, "extremely high-volume" testing.

We all know that Test Pyramid is as follows.

And in my previous article, I mentioned what’s difference between unit tests and integration tests. At the lower levels of the pyramid, the more test cases are required.

Even so, it is difficult to generate a large number of test cases. We usually write corresponding tests based on known conditions or product specifications, sometimes we may remember to write boundary tests (sometimes not), and sometimes we may rely on simple random verification of functionality, e.g. faker.

However, in general, even if we try hard to come up with test cases, we cannot cover all scenarios, and we call this testing method example-based testing. This is because the test cases we come up with are basically extended from a certain example and cannot cover all the unknown contexts nor can we test all the boundary conditions.

At this point, we would like to have a framework automatically generate enough scenarios (reasonable scenarios or not) to verify the code we write, and the test cases we write only need to ensure their "properties" are correct. This is the origin of property-based testing.

Nevertheless

The reality is that integration testing is approximately the same as unit testing.

I have worked in many organizations, from large national enterprises to small startups. Whether I am a developer or a mentor, from past experience, unit testing is about as relevant as integration testing.

For most developers, it is not an easy task to properly divide unit testing and integration testing. To be able to split test cases entirely they need to have the skills of design patterns, dependency injection, dependency inversion, etc. to be able to do it well. Therefore, most test environments are based on a specific test environment, such as using docker-compose to generate a one-time database and test data and test on it.

The documents of fast-check is written based on the standard of unit test, and it seems that only the verification boolean is provided, that is, fc.assert, so I took some time to research to write a test case close to daily use.

Generally I need several abilities.

1. Be able to test async/await.
2. Be able to verify more contexts, such as assertEqual.

fast-check Introduction

Before we start writing test cases, let's have a look at the basic usage of fast-check.

First, let's introduce the structure of fast-check.

• Assertion (fc.assert)
• Properties (fc.property or fc.asyncProperty)

The function of fc.assert is to verify that all the tests automatically generated by the properties are correct. The properties are needed to describe two important blocks.

• Runner
• Arbitraries

Runner is the context to be tested, i.e., the target. On the other hand, the arbitraries are the input parameters of the target, which are automatically generated by the properties, and all we have to do is to provide rules for them, e.g., only integers.

The following is a simple example.

fc.assert(
  fc.property(fc.integer(), fc.integer(), (i, j) => {
    return i + j === add(i, j);
  })
);

The two fc.integer() are arbitraries, and the later anonymous function is the runner, which takes two arguments i and j, corresponding to the previous arbitraries. We want to verify whether the function add really sums the two arguments correctly, so the result of add should be consistent with +.

Let's review the two requirements we just mentioned.

1. fast-check is able to test async/await, runner can be a promise, and fc.assert itself is also a promise.
2. Although our test target is add, but a good integration with some conditions in the runner can make not only the effect of boolean.

fast-check Examples

Now let's come to a more practical example. Suppose I have a database table with money for each user.

user_id	money
123	100
456	200
abc	9999
def	0

There is a function async function getMoney(limit) which will sort money in ascending order and also determine how much money to return based on the parameters.

Now we want to test this black box.

describe("fast-check test", () => {
  before(async () => {
      // generate 10 random records
  });

  it("#1", async () => {
    const result = await getMoney(100);
    expect(result.length).to.be.equal(10);
  });

  it("#2", async () => {
    await fc.assert(
      fc.asyncProperty(fc.integer(), async (i) => {
        const result = await getMoney(i);
        return result.length <= 10 && result.length >= 0;
      })
    );
  });

  it("#3", async () => {
    await fc.assert(
      fc.asyncProperty(fc.integer({ min: 0, max: 10 }), async (i) => {
        const result = await getMoney(i);
        return result.length === i;
      })
    );
  });

  it("#4", async () => {
    await fc.assert(
      fc.asyncProperty(fc.integer(), async (i) => {
        const result = await getMoney(i);
        if (result.length > 1) {
          let prev = parseFloat(result[0]);
          for (let i = 1; i < result.length; i++) {
            const curr = parseFloat(result[i]);
            if (curr < prev) {
              return false;
            }
            prev = curr;
          }
        }
        return true;
      })
    );
  });
});

Let me explain in brief.

1. Just simply verify the function really works, there is no use of fast-check.
2. Given an arbitrary integer, the length of the return result should be between 0 and 10, because we only created ten records in before.
3. Given a range of integers, the length of the return should be equal to the given length.
4. Verify the order of the whole array is indeed ascending. From this runner can be seen, even very complex conditions can be verified, but be careful not to make bugs in the test case resulting in the need for a test case of the test case.

If a problem is detected, fast-check will also tell you what kind of arbitraries it uses to detect the problem. For example,

Counterexample: [-1234567890]

This means the test case failed when i = -1234567890. It is possible the negative number is not handled correctly or the "large" negative number is not handled correctly. This is the time to write a real unit test (or integration test) and verify -1234567890, so that such a failed case can be used as a regression test afterwards.

Conclusion

Ideally, when testing database behavior like this, we would use techniques such as dependency injection to isolate the physical database in order to improve testing performance. But as I said earlier, it is not easy to properly separate code from external dependencies depending on the experience and skill of the developer.

So in many organizations, we still see that most of the test cases have to rely on the physical database for testing. But I have to say this is incorrect.

In this article, I explain the usage of fast-check through a real-life example and how it is close to practice. Nevertheless, I hope we don't have to face this again, at least after reading my previous article, let's try to turn over those unreasonable test cases.