Automated Visual Testing With Robot Framework

A common problem when automating tests are tests which require a visual comparison to a previous state. This can be a very time consuming task when you need to execute many of these testing tasks. But not anymore, the Robot Framework DocTest library comes to the rescue!

1. Introduction

Even if you do not know what Robot Framework is, it is really worth the effort to continue reading. The DocTest library might save you a lot of testing time, but it requires some knowledge of Robot Framework. In case you do not know what Robot Framework is, read some of the previous posts in order to get a good introduction.

• Automated Acceptance Testing With Robot Framework
• How to Write Data Driven Tests With Robot Framework
• Create Custom Robot Framework Libraries

When trying to test your application in an automated way, some tests just cannot be automated. Mainly because these tests require a visual check. Let’s assume that your application makes use of maps which require a regular update, or reports need to be generated and must be verified, etc. These visual tests are probably also very time consuming and there is no guarantee that it is verified correctly. The visual check is done by humans and humans make mistakes. Wouldn’t it be great when you could automate the visual check or at least get notified when something changed compared to a previous version and the output of the test would show you the differences? All of this is now possible with the Robot Framework DocTest Library. It was presented at Robocon 2021. The talk is available online and it will take you only half an hour to watch it.

Enough for the introduction, let’s see how it works. The sources used in this post are available at GitHub.

2. Installation

The installation instructions below are executed with Ubuntu 20.04.2 LTS version. Installation is quite easy using pip:

$ pip install --upgrade robotframework-doctestlibrary

DocTest makes use of other applications, it is necessary to install these otherwise things can go wrong.

$ apt-get install imagemagick
$ apt-get install tesseract-ocr
$ apt-get install ghostscript
$ apt-get install libdmtx0b

If you do not install the above applications, using the DocTest keyword Compare Images will raise the following error:

[ ERROR ] Error in file '/<path>/MyDocTestPlanet/visual_test.robot' on line 2: Importing library 'DocTest.VisualTest' failed: ImportError: Unable to find dmtx shared library

If you want to compare PDF files, ImageMagick might raise the following error from the library:

File could not be converted by ImageMagick to OpenCV Image: testdata/sample_1_page.pdf

Trying to convert a PDF to a JPG file using ImageMagick by means of CLI will give you a more clear error:

$ convert sample.pdf sample.jpg 
convert-im6.q16: attempt to perform an operation not allowed by the security policy

ImageMagick makes use of a policy.xml file which contains by default the following lines which is the cause of the issue:

<!-- disable ghostscript format types -->
<policy domain="coder" rights="none" pattern="PS" />
<policy domain="coder" rights="none" pattern="PS2" />
<policy domain="coder" rights="none" pattern="PS3" />
<policy domain="coder" rights="none" pattern="EPS" />
<policy domain="coder" rights="none" pattern="PDF" />
<policy domain="coder" rights="none" pattern="XPS" />

In the root of the DocTest repository, a policy.xml file is present. Replace the file in /etc/ImageMagick-6/policy.xml file with the one from the repository. The conversion should work now.

3. In Practice

In this section, you will learn how to use the library. Note that the GitHub repository of the library also provides some good examples, which will show you how to use the library. Clone the repository and run the test from the root of the repository:

$ robot atest/Compare.robot

Nevertheless, it is also good to verify whether this will work with items you created yourself. As a base image, some kind of map is created containing a timestamp. All of the images are located in the images directory. The original drawings *.odg files are present and the saved *.png files which will be used in the examples. The base image is base-image.png.

3.1 Comparison With Small Difference

The first comparison you will make, is one with a small difference in one of the triangles (base-image-with-triangle-difference.png). When you would like to compare this visually, you would have a hard time finding the difference because it is such a small one. So, how can we do so by means of the DocTest library? The corresponding test is:

| *** Settings *** |
| Library | DocTest.VisualTest

| *** Test Cases *** |
| 01 | [Documentation] | Compare Two Images With Triangle Difference
|    | Compare Images  | images/base-image.png | images/base-image-with-triangle-difference.png

You import the library and the only thing you need to do, is to run the Compare Images keyword with the reference and canditate image. Run the test:

$ robot visual_test.robot

The test obviously fails because there is a difference. You will notice that a directory screenshots has been created where the diff images are located. But, it is easier to open the log.html file with Robot Framework results.

This result is very useful, you immediately notice where the differences are. As expected, the triangle is indicated as a difference, but also the timestamp. Next to this screenshot, a combined difference image is provided. The results are available in the GitHub repository.

3.2 Comparison With Small Difference Ignoring Timestamp

There is however a problem with the previous comparison. When there are no differences in the map and the screenshots are taken at different points in time, the comparison will still mark the two maps as different. This is probably not what you want. The library provides a solution for this by means of masks. You can define masks in a JSON file where you define which items should be ignored during the comparison. In this case, you create a masks/mask-date.json file as follows:

[
    {
    "page": "all",
    "name": "Date Pattern",
    "type": "pattern",
    "pattern": ".*[0-9]{2}-[0-9]{2}-[0-9]{4}"
    }
]

The pattern being used is a simple regular expression. The test case only needs one extra placeholder_file argument.

| 02 | [Documentation] | Compare Two Images With Triangle difference and Time Mask
|    | Compare Images  | images/base-image.png | images/base-image-with-triangle-difference.png | placeholder_file=masks/masks-time.json

When you run the test now, it still fails because the triangle has a difference, but in the comparison screenshot, the date is marked with blue, meaning that it has been ignored during the comparison.

Now, let’s verify whether the comparison is successful when the images are identical besides the time.

| 03 | [Documentation] | Compare Two Images With Time Mask
|    | Compare Images  | images/base-image.png | images/base-image-with-time-difference.png | placeholder_file=masks/masks-time.json

And yes, the test is now successful although the time in the images is different.

This is great, you now have the possibility to only compare what you want in the image and ignore those items which will be different anyhow.

3.3 Comparison With Small Difference Ignoring Area

Besides a pattern, it is also possible to define an area to be ignored. Create a masks/mask-bottom.json file containing the following content:

[
    {
        "page": "all",
        "name": "Header Area",
        "type": "area",
        "location": "bottom",
        "percent":  10
    }
]

This time, you specify the bottom area and a percentage. The test case is now:

| 04 | [Documentation] | Compare Two Images With Bottom Mask
|    | Compare Images  | images/base-image.png | images/base-image-with-triangle-difference.png | placeholder_file=masks/masks-bottom.json

Running this test shows you that the bottom part of the image is ignored.

3.4 Comparison With Small Move

When an item has moved slightly, the difference will be hard to spot. Besides the expected and actual result image, the Robot Framework log also shows us a combined result. The test is:

| 05 | [Documentation] | Compare Two Images With Small Move
|    | Compare Images  | images/base-image.png | images/base-image-with-small-move.png

In the combined result you can see the difference much better.

What if you would like to accept these kind of small moves? In other words, you do not want to fail the test when an item has moved a little bit. In that case, you can add the move_tolerance argument and specify the amount of pixels you tolerate. The previous test becomes the following:

| 06 | [Documentation] | Compare Two Images With Small Move And Move Tolerance
|    | Compare Images  | images/base-image.png | images/base-image-with-small-move.png | move_tolerance=10

The test now passes successfully and within the result, it is indicated that a small move has been found.

3.5 PDF Compare With Small Difference

Just like images, it is also possible to compare PDF files. This can be extremely useful when you have reports to compare. A small report is available in pdf/base-report.pdf (the original base-report.odt is also available). A report pdf/base-report-with-small-difference.pdf contains a different amount for the screw driver and a different date. The test is similar as for the images, this time you provide the get_pdf_content argument and set it to true.

| 07 | [Documentation] | Compare Two PDF Files With Small Difference
|    | Compare Images  | pdf/base-report.pdf | pdf/base-report-with-small-difference.pdf | get_pdf_content=${true}

Again, the log file shows you exactly the difference:

Masks also work with this kind of comparison. Let’s apply the date mask.

| 08 | [Documentation] | Compare Two PDF Files With Small Difference and Date Mask
|    | Compare Images  | pdf/base-report.pdf | pdf/base-report-with-small-difference.pdf | placeholder_file=masks/masks-date.json | get_pdf_content=${true}

As expected, only the amount difference is shown and the date is marked with blue indicating that it has been ignored.

4. Conclusion

The Robot Framework DocTest library is a really powerful library to compare differences in images and reports. It is simple in its use and it works just fine. This will definitely become a big time saver when you apply this during your testing efforts and it will reduce the chance for testing errors.