I am big fan of allowlist-testing. I honestly think allowlist-testing is the future of testing for testing any interface. For those unfamiliar with the term: where typical assertion-based testing is denylist-testing (denying specified changes, ignores all else), allowlist-testing guards against all changes, except for the changes you specified as irrelevant, i.e. allowlisted. Other names for that technique are whitelist/blacklist testing (now dismissed for being racist), difference testing, snapshot testing, Golden Master testing, approval testing or characterization testing.
I wanted to come up with an executable demo of a tool with Allowlist testing capabilities. It should allow you to focus on the concept and to play around with it to get some first impression, with the least amount of overhead possible. Such a demo can be used in workshops and conferences, it can be tweeted and otherwise easily shared. For that, there are 3 simple requirements:
- The tool should be hassle-free and simple to install and play around with, to produce results with as little overhead and as little prerequisites in terms of knowledge and technology as possible.
- The tool should demo the many advantages of allowlist-testing.
- It should be vendor-agnostic, i.e. the tool should ideally be open source and to the least extent commercial.
This list of requirements is short and relatively straight forward, and there are many implementations for allowlist-testing. Yet in my opinion, candidates fulfilling those criteria are hard to come by with.
Hassle-free and simple to try
That the tool should be hassle-free and simple to try disqualifies quite a bunch of interesting candidates. Most open source frameworks are test frameworks that are used during development by developers. Which means that you need to set up a project environment with some build mechanism for test execution.
Jest
Being used in over 1.000.000 public repos on GitHub, Jest (https://github.com/facebook/jest) is a very popular example of allowlist-testing—if not the most popular example. However, it only works with JavaScript in a Node, React or Angular project environment. And setting up yarn or node.js together with any such project just to execute some tests for a short demo is definitely neither simple nor hassle-free.
Approval Tests
The same applies to another great example of allowlist-testing, which is Approval Tests (https://approvaltests.com/). Approval Tests is available as a library in almost any platform (Java, C++, NodeJS, Python, …). But it is mainly focusing on technical interfaces, and works great with XML and other technical formats. The barriers to using it are definitely a lot lower than with Jest. You only need a test that can be executed in any of the supported platforms, e.g. in Java. It is open source and free to use, so approval tests also fulfills the other criteria.
TextTest
Another great tool is TextTest (http://texttest.sourceforge.net) that runs on Python and can be combined with a number of other platforms, including Java Swing, SWT and Tkinter. To set it up, you need to install Python and some more, depending of the platform of your choice. However, it is very much geared towards domain-language-written tests. For a more visual test of e.g. a GUI (custom or Web) it needs some more tooling to drive the GUI. For Python and Java GUIs there is StoryText, which is especially designed to work with TextTest.
recheck-web
recheck-web (https://github.com/retest/recheck-web) is open source and comes with a Chrome extension that can be used to easily try it. A Chrome extension is both simple to install and simple to remove. In order for the Chrome extension to run without any additional setup costs, it sends the data to retest.org. To guard your sensible test data, you need to create a free account before trying it. The detailed results are in a proprietary format, that you need an open source CLI or a free GUI to open. The GUI comes in a self-contained ZIP file.
Demo the Advantages of Allowlist-testing
Although pixel comparison is a way of Golden Master testing, a tool must implement some mechanism to allowlist changes (i.e. not notifying the user when they occur), in order to also count as a allowlist-testing tool. This is important, so the tool can be used in test automation on a regular basis, without reporting (too many) false positives. Many tools fail at this, such as
- Appraise (https://github.com/AppraiseQA/appraise)
- Screenster.io (https://screenster.io)
- Diff.io (https://diff.io)
- Shutter (https://shutter.sh)
In the long term, mere pixel-comparison is of limited value. A good tool should both allow to mass-accept a change, as allowlist-testing often creates redundancies, and should provide convenient ignore options.
Be Open source and not Commercial
The remainder of the tools are commercial, none offer an open source solution to their testing tool:
- Screener.io (https://screener.io)
- Applitools (https://applitools.com)
- Percy (https://percy.io)
- Mabl (https://www.mabl.com/visual-change-detection)
Summary
So, what do you choose when you want to demonstrate allowlist-testing to a bunch of testers, that have diverse backgrounds (i.e. are used to work on different platforms)? Do you happen to know any other tool option we forgot?
Top comments (0)