Scaling visual testing with BackstopJS (Part 2)

In Part 1, I wrote about how I break up the monolithic BackstopJS config file. We were able to organize multiple test files in folders any way we liked while at the same time reducing duplication between scenarios. This dealt with the first of two related limitations of BackstopJS: one giant flat configuration file. Now, in Part 2, we’ll deal with the second limitation: one giant flat report.

The problem

For all you test tool developers out there, take this note: If your product outputs a single page report of all the test results with no hierarchy and no way of navigating to different subsections of tests whatsoever, your tool doesn’t scale. Full stop.

While Backstop does allow filtering results by either pass/fail or text search, and has navigation buttons for skipping up or down to the next scenario, it doesn’t have any way of grouping tests together. Much like this flat format creates an unmanageably long list of scenarios in the config file, it also creates a very long list of results in the output report. With thumbnails turned off, it looks like this:

I may be able to see which specific test fails, but it is often just as useful to see which ones didn’t fail. That context helps me understand what the scope of the issue is likely to be.

So, our goal is to add a way that we can group tests together and navigate or filter the report page to see the results for just that subset of tests.

Backstop’s HTML report is actually a React app, so adding new functionality to it isn’t as easy as a wrapper script like we did for Part 1. I’m not going to walk through the development as closely this time, since you can find my complete code in a fork on github. Here are the basic components.

Parsing a hierarchy from labels

When we combined our small scenario files into a single Backstop, we included the filename in the label field of the scenario.

theseScenarios.forEach(scenario => {
    labelPrefix = filename.replace(/^\.\//, "").replace(/^\//, "");
    scenario.label = labelPrefix + "/" + scenario.label;
});

This meant we didn’t have to worry about duplicate labels across different files, but now it gives us a natural way of grouping tests.

Backstop maintains a list of visible tests in the state, so we pull this into a new component:

const mapStateToProps = state => {
  return {
    tests: state.tests.filtered
  }
};

And then we can reconstruct where in our directory structure each test came from:

const labels = this.props.tests.map(test => test.pair.label);
const tree = labels.reduce((tree, label) => {
    const parts = path.split('/');
    let currentNode = tree;
    parts.forEach((part, index) => {
        if (!currentNode.hasOwnProperty(part)) {
            currentNode[part] = {
                label: part,
                nodes: {}
            };
        }
        currentNode = currentNode[part].nodes;
    });
    return tree;
}, {});

This breaks the label up using the “/” character (since we know that’s how we created these labels in the first place) and then adds each part of the label as one node in a tree. Assuming the same directory structure from Part 1, the resulting structure should look something like this:

{
    "tests": {
        "homepage.js": {
            "header": {}
            "main body": {}
            ...
        },
        "profile.js": {
             ...
        },
        "widgets": {
            "one.js": { ... },
            "two.js": { ... },
        },
        "gizmos": {
            "alpha.js": { ... },
            "beta.js": { ... },
        }
    }
}

where I’ve abbreviated lists of individual scenarios into ellipses. This is definitely something that could be simplified if this style of config files was incorporated into the Backstop core, but this does the job. With this structure, we can then develop a UI that lets us navigate it.

Sketching out a UI

I decided the simplest thing to do would be to add a new section to the report that allowed changing the Backstop report app’s filter properties. We need:

• A new action/reducer, NAVIGATE_TESTS, that takes a point in the label hierarchy and change’s the app’s state.filtered list to show only the tests under that node.
• A button component for the UI, NavButton, to allow the user to select a point in the hierarchy, firing the above action.
• A Navigation component to render a navigable hierarchy that we could put into the report.
• A way of displaying how many passes and fails are under each hierarchy node, so the user knows where to look, which we’ll call the PassMeter.

While I’m sure there are lots of clever ways to implement a UI for what is effectively folders and files, I kept it quick and simple as much as possible by reusing existing UI components. Each NavButton ended up being a condensed version of the TestCard components that are used to display the test results themselves, for example, and I don’t have any fancy animations for expanding/collapsing sections.

The main question was finding a balance between displaying enough information that the user can get the context they need, without getting lost in a potentially huge hierarchy of tests. I decided to show breadcrumbs for each point in the hierarchy so the user could also navigate back up the tree, and only expand nodes under the current one. When first opening the report, you would see just the top level:

tests
|- homepage.js
|- profile.js
|- gizmos
|- widgets

Clicking on “homepage.js” will show only the tests that were defined in that config file:

tests
|- homepage.js
    |- header
    |- main body
    |- ...

I found it useful to include the individual test labels in the hierarchy here because each one can still represent multiple tests if there are multiple viewports in effect. Even if not, it still ends up being a nicely condensed version of the test results.

If instead you clicked on the “gizmos” folder:

tests
|- gizmos
    |- alpha
    |- beta

Clicking on the root, “tests”, would go back to the previous state, by firing off the same action as any other nav button, setting the filter to all tests under tests/*. For the UI components, we actually have two steps. First, we output those higher-level breadcrumbs if there are any:

let currentNode = tree;
let depth = 0
while (Object.keys(currentNode).length === 1) {
    const key = Object.keys(currentNode)[0];
    const node = currentNode[key];
    breadcrumbs.push(buttonFor(node, depth));
    currentNode = node.nodes;
    depth++;
}

In this snippet, buttonFor is going to return the navigation button for that point in the hierarchy, and depth is keeping track of where we are just so we can style the nesting appropriately later. Since we constructed the tree object based only on what tests are visible, this is using the trick that we’ll only have multiple child nodes if all of those nodes are included in the currently active filter. As soon as we hit a point with multiple child nodes, we’re done with the breadcrumbs. This has the nice side benefit that if you have a string of tests nested under each other without any branching, you won’t have to click down into every sub node to get to the interesting stuff.

Then, we just wrap it out by listing the possible nodes below the currently selected one, at whatever depth we left off at:

for (const key in currentNode) {
    const node = currentNode[key];
    subNodes.push(buttonFor(node, depth));
}

The only detail at that point is making sure that the navigation buttons dispatch our filter action. I readily admit that this isn’t a particular robust or well architected filtering system, which is why I haven’t tried to push it into Backstop core yet, but it does the job for us. The end result looks something like this:

And again, the power here is that when something goes wrong, I have a more immediate sense of where exactly the problem lies:

The gizmos and profile sections are fine, just the homepage and widgets need to be looked into further.

Installing the new report

Until the built in reporting UI gets some navigation enhancements (or I get enough time to polish up this hack into a serviceable pull request myself), we have to take extra steps to get this improved version when running Backstop.

One option was to install Backstop from our own fork with these UI changes, but I wanted to avoid the extra maintenance on keeping up with non-UI changes that Backstop might make. Instead, I decided to only “fork” the UI portion of the code. To do this, we’re going to add an extra to the script we created in the previous part. It will now do three things:

1. Compile separate config files into a single BackstopJS config.
2. Run Backstop itself, either in test or reference mode (or both).
3. Overwrite the default UI with our improved one.

Conveniently, the Backstop UI compiles into a single JavaScript file, so this new step is just a simple copy:

const reportPath = path.join(
    process.cwd(),
    baseConfig.paths.html_report
);
fs.copyFile(
    path.join(__dirname, "report/index_bundle.js"),
    path.join(reportPath, "index_bundle.js"),
    error => console.error(error)
);

This takes our build, which we saved in report/index_bundle.js, and copies it to the html_report path defined in our base BackstopJS config. Doing this automatically whenever this helper script is runs means we’ll always get the enhanced report.

And that’s all there is to it. Despite having glossed over the details of the React components, I hope this gives some direction in how one can tweak an open source tool like this to overcome some of its shortcomings. If you want to dig deeper and play with extending this yourself, I have this code in a fork on github (view the changes here). Again, this isn’t perfect or optimal in anyway, but if you’re keep on contributing to Backstop itself, consider this a proof of concept for making the reporting more scalable.