DEV Community: Michael Battat

How to eradicate visual drift in applications

Michael Battat — Tue, 16 Mar 2021 16:52:28 +0000

Recently we released a case study with Sonatype that showed how it eradicated visual drifts in its applications. If you're interested, here's what we shared.

The Situation

Sonatype first bought Applitools in 2017 to address a core problem affecting web applications: visual drift due to code changes with unaccounted UI dependencies. In developing the Nexus Lifecycle product, Sonatype learned that small code changes could change the application visually in unanticipated ways.

Nexus runs on developer systems that include Windows, Mac, and Linux. Their customers would run various versions of browsers – from most stable to most recent. The big driver for Sonatype involved ensuring Internet Explorer 8 support even as development moved beyond IE 8 to more recent browser versions.

Multiple development teams worked on the product. Teams would reuse each other’s code. Small changes in markup or function used by one team might impact the code of another. Sonatype used a range of functional automation tools, but visual issues went undetected. Understanding the limitations of functional tests to catch visual defects, developers had to devote engineering time to uncover these failing cases manually.

Jamie Whitehouse and everyone on the development team spent time each release working to uncover and address these undetected failures. Often, this work occurred as spot checks of the 1,000+ pages of the application during development. In reality, this work, and the inherent risk of unintended changes, slowed the delivery of the product to market.

Looking For A Solution

Jamie and his team sought an approach for highlighting these differences. Key capabilities he needed:

Automated – a solution that could capture and highlight visual differences without manual intervention
Accurate – low incidence of false positives and missed errors. Sonatype needed element-level accuracy, and a way to ignore pixel-level differences that plague pixel diff tools.
Integrated – easy to integrate with existing tests.
Complete – In evaluating tools, Jamie wanted to avoid tools that would require his team’s incremental development work to use. Tooling work would cost development hours without delivering value to Sonatype customers.
Supported – Sonaytpe would pay for support instead of using their own engineers to support the solution.

Jamie’s team investigated the available solutions. They wrote off pixel-diff comparators because the team’s experience indicated a high degree of false positives from pixel diffs. Also, the Sonatype use case differed greatly from graphic design use cases, where pixel match matters. They also wrote off open source tools, which put engineering resources at risk to develop and maintain a solution that met Sonatype’s needs.

Given the constraints, Jamie says, Applitools was the obvious choice to try. Applitools provided AI-based approach that allowed it to identify and compare visual elements instead of comparing pixels. It also provided a range of match comparisons, including layout comparisons, content checks, and even exact pixel comparison (if needed). Jamie’s team saw the promise of automated visual validation without false positives.

During their initial deployment, Applitools met their needs. It integrated with their existing test automation, delivered accurate results, and Applitools provided great support.

Results

After two years of using Applitools, Sonatype has reaped some serious rewards.

While they continue to justify the cost of Applitools based on the engineering effort saved, the real benefit involves the complete elimination of visual drift in their applications. Sonatype now knows, with certainty, the effect of changes to rendering and markup across reused components – simply through running their test automation. Developers can deliver code changes to existing code and know that Applitools can properly catch all the changes – including unintended changes – early in the development process.

If Sonatype engineers make a change in their margins across a number of pages, all the differences show up as highlights in Applitools. Features in Applitools like Auto-Maintenance make visual validation a time saver. Auto-Maintenance lets engineers quickly accept identical changes across a number of pages – leaving only the unanticipated differences. As Jamie says Applitools takes the guesswork out of testing the rendered pages.

As Jamie explains:

“With unit tests, you check them in once and then run them forever. Once they’re running, you expect them to stay green and check them only when they’re red. Now, imagine being able to do the same thing with UI tests. You write them once and only check them when they’re red. Imagine how much more productive you would be if you didn’t have to worry about missing visual changes?”

With Applitools deployed, Jamie has his engineering team focused on delivering value in Nexus Lifecycle. Applitools has helped uncover countless unexpected changes during the development phase, as opposed to waiting for final testing. The true value of using Applitools – development free from visual risk – is hard to quantify. But, when calculating the speed of delivering products to market, Jamie knows he’s much faster with Applitools.

Where Next?

To reduce its UI risk further, the Sonatype team is deploying a shared components library. Their library will form the basis of the future UI, with components reused across their Nexus Lifecycle application. The component team evaluated Applitools alongside several solutions specifically geared toward component testing. They concluded that Applitools would test their entire component library with the same accuracy and workflow they rely on in their existing end-to-end tests.

Sonatype plans to use Applitools to test the entire library in action across all browsers and viewports. Their goal is to deliver visual validation even earlier in the development phase and diminish their visual risk in end-to-end tests.

Jamie expects this approach can speed their delivery further, as Sonatype can re-use components with much lower risk of unexpected UI behavior.

Get A Jump Into GitHub Actions

Michael Battat — Thu, 04 Mar 2021 20:11:28 +0000

On January 27, 2021, Angie Jones of Applitools hosted Brian Douglas, aka “bdougie”, Staff Developer Advocate at GitHub, for a webinar to help you jump into GitHub Actions. You can watch the entire webinar on YouTube.

What Are GitHub Actions?

Angie’s first question asked Brian to jump into GitHub Actions.

Brian explained that GitHub Actions is a feature you can use to automate actions in GitHub. GitHub Actions let you code event-driven automation inside GitHub. You build monitors for events, and when those events occur, they trigger workflows.

If you’re already storing your code in GitHub, you can use GitHub Actions to automate anything you can access via webhook from GitHub. As a result, you can build and manage all the processes that matter to your code without leaving GitHub.

Build Test Deploy

Next, Angie asked about Build, Test, Deploy as what she hears about most frequently when she hears about GitHub Actions.

Brian mentioned that the term, GitOps, describes the idea that a push to GitHub drives some kind of activity. A user adding a file should initiate other actions based on that file. External software vendors have built their own hooks to drive things like continuous integration with GitHub. GitHub Actions simplifies these integrations by using native code now built into GitHub.com.

Brian explained how GitHub Actions can launch a workflow. He gave the example that a team has created a JavaScript test in Jest. There’s an existing test using Jest – either npm test, or jest. With GitHub Action workflows, the development team can automate actions based on a starting action. In this case, the operator can drive GitHub to execute the test based on uploading the JavaScript file.

Get Back To What You Like To Do

Angie pointed out that this catchphrase, “Get back to what you like to do,” caught her attention. She spends lots of time in meetings and doing other tasks when she’d really just like to be coding. So, she asked Brian, how does that work?

Jump into GitHub Actions and get back to programming work.
Brian explained that, as teams grow, so much more of the work becomes coordination and orchestration. Leaders have to answer questions like:

What should happen during a pull request?
How do we automate testing?
How do we manage our build processes

When engineers have to answer these questions with external products and processes, they stop coding. With GitHub Actions, Brian said, you can code your own workflow controls. You can ensure consistency by coding the actions yourself. And, by using GitHub Actions, you make the processes transparent for everyone on the team.

Do you want a process to call Applitools? That’s easy to set up.

Brian explained that GitHub hosted a GitHub Actions Hackathon in late 2020. The team coded the controls for the submission process into the hackathon. You can still check it out at githubhackathon.com.

The entire submission process got automated to check for all the proper files being included in a submission. The code recognized completed submissions on the hackathon home page automatically.

Brian then gave the example of work he did on the GitHub Hacktoberfest in October. For the team working on the code, Brian developed a custom workflow that allowed any authenticated individual to sign up to address issues exposed in the Hackathon. Brian’s code latched onto existing authentication code to validate that individuals could participate in the process and assigned their identity to the issue. As the developer, Brain built the workflow for these tasks using GitHub Actions.

What can you automate? Informing your team when a user does a pull request. Send a tweet when the team releases a build. Any webhook in GitHub you can automate with GitHub Actions. For example, you can even automate the nag emails that get sent out when a pull request review does not complete within a specified time.

Common Actions

Angie then asked about the most common actions that Brian sees users running.

Brian summarized by saying, basically, continuous integration (CI). The most common use is ensuring that tests get run against code as it gets checked in to ensure that test suites get applied. You can have tests run when you push code to a branch, push code to a release branch or do a release, or even when you do a pull request.

While test execution gets run most frequently, there are plenty of tasks that one can automate. Brian did something specific to assign gifts to team members who reviewed pull requests. He also used a cron job to automate a GitHub Action which opened up a global team issue each Sunday US, which happens to be Monday in Australia, and assigned all the team members to this issue. Each member needed to explain what they were working on. This way, the globally-distributed team could stay on top of their work together without a meeting that would occur at an awkward time for at least one group of team members.

Brian talked about people coming up with truly use cases – like someone linking IOT devices to webhooks in existing APIs using GitHub Actions.

But the cool part of these actions is that most of them are open source and searchable. Anyone can inspect actions and, if they don’t like them, modify them. If a repo includes GitHub Actions, they’re searchable.

On github.dom/bdougie, you can see existing workflows that Brian has already put together.

Jump Into GitHub Actions – What Next?

I shared some of the basic ideas in Brian’s conversation with Angie. If you want to jump into GitHub Actions in more detail, you can check out the full webinar and the slides in Addie Ben Yehuda’s summary blog for the webinar. That blog also includes a number of Brian’s links, several of which I include here as well:

https://bdougie.live/
https://github.com/bdougie/live
https://lab.github.com/githubtraining/devops-with-github-actions
https://youtube.com/ilikerobot
https://github.com/bdougie
https://docs.github.com/en/actions/reference/events-that-trigger-workflows

Enjoy jumping into GitHub Actions!

How Thunderhead Speeds Quality Delivery with Applitools

Michael Battat — Wed, 17 Feb 2021 17:23:15 +0000

Thunderhead is the recognised global leader in the Customer Journey Orchestration and Analytics market. The ONE Engagement Hub helps global brands build customer engagement in the era of digital transformation.

Thunderhead provides its users with great insights into customer behavior. To continue to improve user experience with their highly-visual web application, Thunderhead develops continuously. How does Thunderhead keep this visual user experience working well? A key component is Applitools.

Before – Using Traditional Output Locators

Prior to using Applitools, Thunderhead drove its UI-driven tests with Selenium for browser automation and Python as the primary test language. They used traditional web element locators both for setting test conditions and for measuring the page responses.

Element locators have been state-of-the-art for measuring page response because of precision. Locators get generated programmatically. Test developers can find any visual structure on the page as an element.

Depending on page complexity, a given page can have dozens, or even hundreds, of locators. Because test developers can inspect individual locators, they can choose which elements they want to check. But, locators limit inspection. If a change takes place outside the selected locators, the test cannot find the change.

These output locators must be maintained as the application changes. Unmaintained locators can cause test problems by reporting errors because the locator value has changed while the test has not. Locators may also remain the same but reflect a different behavior not caught by a test.

Thunderhead engineers knew about pixel diff tools for visual validation. They also had experience with those tools; they had concluded that pixel diff tools would be unusable for test automation because of the frequency of false positives.

Introducing Applitools at Thunderhead

When Thunderhead started looking to improve their test throughput, they came across Applitools. Thunderhead had not considered a visual validation tool, but Applitools made some interesting claims. The engineers thought that AI might be marketing buzz, but they were intrigued by a tool that could abstract pixels into visible elements.

As they began using Applitools, Thunderhead engineers realized that Applitools gave them the ability to inspect an entire page. Not only that, Applitools would capture visual differences without yielding bogus errors. Soon they realized that Applitools offered more coverage than their existing web locator tests, with less overall maintenance because of reduced code.

The net benefits included:

Coverage – Thunderhead could write tests for each visible on-page element on every page.
Maintainability – By measuring the responses visually, Thunderhead did not have to maintain all the web element locator code for the responses – reducing the effort needed to maintain tests.
Visual Validation – Applitools helped Thunderhead engineers see the visual differences between builds under test, highlighting problems and aiding problem-solving.
Faster operation – Visual validation analyzed more quickly than traditional web element locators.

Moving Visual Testing Into Development

After using Applitools in end-to-end testing, Thunderhead realized that Applitools could help in several areas.

First, Applitools could help with development. Often, when developers made changes to the user interface, unintended consequences could show up at check-in time. However, by waiting for end-to-end tests to expose these issues, developers often had to stop existing work and shift context to repair older code. By moving visual validation to check-in, Thunderhead could make developers more effective.

Second, developers often waited to run their full suite of element locator tests until final build. These tests ran against multiple platforms, browsers, and viewports. The net test run would take several hours. The equivalent test. using Applitools took five minutes. So, Thunderhead could run these tests with every build.

For Thunderhead, the net result was both greater coverage with tests run at the right time for developer productivity.

Adding Visual Testing to Component Tests

Most recently, Thunderhead has seen the value of using a component library in their application development. By standardizing on the library, Thunderhead looks to improve development productivity over time. Components ensure that applications provide consistency across different development teams and use cases.

To ensure component behavior, Thunderhead uses Applitools to validate the individual components in the library. Thunderhead also tests the components in mocks that demonstrate the components in typical deployment uses cases.

By adding visual validation to components, Thunderhead expects to see visual consistency validated much earlier in the application development cycle.

Other Benefits From Applitools

Beyond the benefits listed above, Thunderhead has seen the virtual elimination of visual defects found through end-to-end testing. The check-in and build tests have exposed the vast majority of visual behavior issues during the development cycle. They have also made developers more productive by eliminating the context switches previously needed if bugs were discovered during end-to-end testing. As a result, Thunderhead has gained greater predictability in the development process.

In turn, Thunderhead engineers have gained greater agility. They can try new code and behaviors and know they will visually catch all unexpected behaviors. As a result, they are learning previously-unexplored dependencies in their code base. As they expose these dependencies, Thunderhead engineers gain greater control of their application delivery process.

With predictability and control comes confidence. Using Applitools has given Thunderhead increased confidence in the effectiveness of their design processes and product delivery. With Applitools, Thunderhead knows how customers will experience the ONE platform and how that experience changes over time.

"Our quality increases exponentially with Applitools. We run it with every build - which takes around five minutes. Without Applitools - the process would take 4 hours per build for less coverage than what we do now in 5 minutes - simply not something we could afford."

WALT HARRIS — Head of Quality at Thunderhead

To read more about the technical details and how Thunderhead's engineering team speeds quality software delivery with Applitools, check this out!

Five Data-Driven Reasons To Add Visual AI To Your End-To-End Tests

Michael Battat — Tue, 19 May 2020 17:08:51 +0000

Do you believe in learning from the experiences of others? If others found themselves more productive adding Visual AI to their functional tests, would you give it a try?

In November 2019, over 3,000 engineers signed up to participate in the Applitools Visual AI Rockstar Hackathon. 288 completed the challenge and submitted tests – comparing their use of coded test validation versus the same tests using Visual AI. They found themselves with better coverage, faster test development, more stable test code, with easier code test code maintenance.

On April 23, James Lamberti, CMO at Applitools, and Raja Rao DV, Director of Growth Marketing at Applitools, discussed the findings from the Applitools Hackathon submissions. The 288 engineers who submitted their test code for evaluation by the Hackathon team spent an average of 11 hours per submission. That’s over 3,000 person-hours – the equivalent of 1 ½ years of engineering work.

Over 3000 participants signed up. They came from around the world.

They used a variety of testing tools and a range of programming languages.

In the end, they showed some pretty amazing results from adding Applitools Visual AI to their existing test workflow.

Describing the Hackathon Tests

Raja described the tests that made up the Hackathon.

Each test involved a side-by-side comparison of two versions of a web app. In one version, the baseline, the page rendered correctly. In the other version, the new candidate, the page rendered with errors. This would simulate the real-world issues of dealing with test maintenance as apps develop new functionality.

Hackathon participants had to write code that did the following:

Ensure the page rendered as expected on the baseline.
Capture all mistakes in the page rendering on the new candidate
Report on all the differences between the baseline and the new candidate

Also, Hackathon participants needed to realize that finding a single error on a page met the necessary – but not sufficient condition for testing. A single test that captures all the problems at once has a faster resolution time than running into multiple bug capture/fix loops. Test engineers needed to write tests that captured all the test conditions, as well as properly reporting all the failures.

Hackathon participants would code their test using a conventional test runner plus assertions of results in the output DOM. Then, they used the same test runner code but replaced all their assertions with Applitools Visual AI comparisons.

To show these test results, he used the Github repository of Corina Zaharia, one of the platinum Hackathon winners.

At this point here, Raja walked through each of the test cases.

CASE 1 – Missing Elements

Raja presented two web pages. One was complete. The other had missing elements. Hackathon participants had to find those elements and report them in a single test.

To begin coding tests, Corina started with the baseline. She identified each of the HTML elements and ensured that their text identifiers existed. She wrote assertions for every element on the page.

In evaluating submissions, judges ensured that the following differences got captured:

The title changed
The Username icon was missing
The Password icon was missing
The username placeholder changed
The wrong password label
The password placeholder changed
There was extra space next to the check box
The Twitter icon had moved
The Facebook icon had moved
The LinkedIn Icon was missing.

Capturing this page required identifying element locators and validating locator values.

In comparison, adding Visual AI required only three instructions:

Open a capture session
Capture the page with an eyes.checkWindow() command
Close the capture session

No identifiers needed – Applitools captured the visual differences.

With much less coding, Applitools captured all the visual differences. And, test maintenance takes place in Applitools.

CASE 2 – Data-Driven Testing

In Case 2, Hackathon participants needed to validate how a login page behaved when applying different inputs. The test table looked like this:

No username, no password
Username, no password
Password, no username
Username and password combination invalid
Valid username and password

Each condition resulted in a different response page.

Hackathon participants found an identical page to the tests in Case 1 – but they were responsible for handling the different responses to each of the different test conditions.

Again, the coding for the conventional test required entering the test conditions via test runner asserting all the elements on the page, including asserting error messages.

Also, the question was left open for testers – what should they test when they test the valid password and username condition? The simplest answer – just make sure the app visits the correct target post-login page. But, more advanced testers wanted to make sure that the target paged rendered as expected.

So, again, the comparison with coded assertions and adding Visual AI makes clear how much more easily Visual AI captures baselines and then compares the new candidate against the baselines.

CASE 3 – Testing Table Sort

The next case – testing table capabilities – fits into capabilities found on many web apps that provide multiple selections. Many consumer apps, such as retailers, reviewers, and banks, provide tables for their customers. Some business apps provide similar kinds of selectors – in retail, financial, and medical applications. In many use cases, users expect tables with advanced capabilities, such as sorting and filtering.

Tables can provide some challenges for testers. Tables can contain lots of elements. Many table functions can require complex test coding – for example, sorting and filtering.

To test table sorting with conventional assertion code, Hackathon participants had to write code that captured all the data in the table, performed the appropriate sort of that data, and use the internally-sorted table in the test code with the sorted table on the web page. Great test coders took pains to ensure that they had done this well and could handle various sorting options. The winners took time to ensure that their code covered the table behavior. This complex behavior did not get caught by all participants, even with a decent amount of effort.

In contrast, all the participants understood how to test the table sort with Visual AI. Capture the page, execute the sort, capture the result, and validate inside Applitools.

Case 4 – Non-Textual Plug-ins

The fourth case involved using graphical rendering of a table in canvas. How do you test that?

Without normal web element locators, a lot of participants got lost. They weren’t sure how to start finding the graphing elements and to build a comparison between the baseline behavior and the new candidate.

Winning Hackathon participants dug into the rendering code to find the javascript calls for the graph and the source data for table elements. This allowed them to extract the values that should be rendered and compare them between the baseline and the new candidate. And, while the winners wrote fairly elegant code, this particular skill took time to dive into JavaScript. And, a fair amount of coding effort.

As with the table sorting Case 3, all the participants understood how to test the graph with Visual AI. Capture the page, and then compare the new candidate with the baseline in Applitools.

Case 5 – Dynamic Data

The final case required the participants to test a page with floating advertisements that can change. In fact, as long as content gets rendered in the advertising box, and the rest of the candidate remains unchanged, the test passes.

The winning participants coded conditional tests to ensure that code existed in the advertising boxes, though they did not have the ability to see how that code got rendered.

With Visual AI, participants had to use different visual comparison modes in Applitools. The standard mode – Strict Mode – searches for visual elements that have moved or rendered in unexpected ways. With dynamic data, Strict Mode comparisons fail.

For these situations, Applitools offers Layout Mode instead. When using Layout Mode, the text and graphical elements need to share order and orientation, but their actual visual representation can be different. In Layout Mode, the following are considered identical – image above text.

However, the pair below has a different layout. On the left, the text sits below the image, while on the right the text sits above the image

Applitools users can hard-code their check mode for different regions into their page capture. Alternatively, they can use Strict Mode for the entire page and handle the region as a Layout Mode exception in the Applitools UI.

All the Hackathon participants, whether coding their tests for Layout mode for the region or by using Layout mode for the selected area once the baseline had been captured in Applitools, had little difficulty coding their tests.

Learning From Hackathon Participants

At this point, James began describing what we had learned from the 1.5 person-years of coding work done on the Hackathon. We learned what gave people difficulty, where common problems occurred, and how testing with Visual AI compared with conventional assertions of values in the DOM.

Faster Test Creation

I alluded to it in the test description, but test authors wrote their tests much more quickly using Visual AI. On average, coders spent 7 person-hours writing coded assertion-based tests for the Hackathon test cases. In contrast, they spent a mere 1.2 hours writing tests using Visual AI for the same test cases.

Interestingly, the prize-winning submitters spent, on average 10.2 hours writing their winning submissions. They wrote more thorough conventional tests, which would yield accurate coverage when failures did occur. On the other hand, their coverage did not match the complete-page coverage they got from Visual AI. And, their prize-winning Visual AI tests required, on average, six minutes more to write than the average of the whole of the test engineers.

More Efficient Coding

The next takeaway came from calculating coding efficiency. For conventional tests, the average participant wrote about 350 lines of code. The prize winners, whose code had greater coverage, wrote a little more than 450 lines of code, on average. This correlates with the 7 hours and 10 hours of time spent writing tests. It’s not a perfect measure, but participants writing conventional tests wrote about 50 lines of code per hour over 7 hours, and the top winners wrote about 45 lines of code per hour over 10 hours.

In contrast, with Visual AI, the average coder needed 60 lines of code, and the top coders only 58 lines of code. Visual AI still results in 50 lines of code per hour for the average participant, and 45 lines of code for the winning participant. But, they are much more efficient.

More Stable Code

End-to-end tests depend on element locators in the DOM to determine how to apply test conditions, such as by allowing test runners to enter data and click buttons. Conventional tests also depend on locators for asserting content in the response to the applied test conditions.

Most software engineers realize that labels and other element locators get created by software developers – who can change locators due to intentional change or unanticipated difference. And element locator using Xpath can suddenly discover the wrong relative locator due to an enhancement. The same is true for labels, which can change between releases – even when there is no visible user behavior difference.

No one wants testing to overconstrain development. No one wants development to remain ignorant of testing needs. And yet, because mistakes sometimes happen, or changes are sometimes necessary, locators and labels change – resulting in test code that no longer works properly.

Interestingly, when evaluating conventional tests, the average Hackathon participant used 34 labels and locators, while the Hackathon prize winners used 47 labels and locators.

Meanwhile, for the Visual AI tests, the average participant used 9 labels and locators, while the winning submissions used only 8. At a conservative measure, Visual AI reduces the dependency of code on external factors – we calculate it at 3.8 x more stable.

Catching Bugs Early

Visual AI can catch bugs early in coding cycles. Because Visual AI depends on the rendered representations and not on the code to be rendered, Visual AI will catch visual differences that might be missed by the existing test code. For instance, think of an assertion for the contents of a text box. In this new release, the test passes because the box has the same text. However, the box width has been cut in half, causing the text to extend outside the box boundary and be obscured. The test passes, but in reality it fails. The test assumed a condition that is no longer true.

Visual AI catches these differences. It will catch changes that result in different functional behavior that requires new coding. It will catch changes – like the one described above, that result in visual differences that impact users. And, it will avoid flagging changes that may change the DOM but not the view or behavior from the user’s perspective.

Easier to Learn than Code-Based Testing

The last thing James shared involved the learning curve for users. In general, we assumed that test coverage and score on the Hackathon evaluation correlated with participant coding skill. The average score achieved by all testers using conventional code-based assertions was 79%. After taking a 90-minute online course on Visual AI through Test Automation University, the average score for Visual AI testers was 88%.

Because people don’t use visual capture every day, testers need to learn how to think about applying visual testing. But, once the participants had just a little training, they wrote more comprehensive and more accurate tests, and they learned how to run those test evaluations in Applitools.

What This Means For You

James and Raja reiterated the benefits they outlined in their webinar: faster test creation, more coverage, code efficiency, code stability, early bug catching and ease of learning. Then they asked: what does this mean for you?

If you use text-based assertions for your end-to-end tests, you might find clear, tangible benefits from using Visual AI in your product release flow. It integrates easily into your CICD or other development processes. It can augment existing tests, not requiring any kind of rip and replace. Real, tangible benefits come to many companies that deploy Visual AI. What is stopping you?

Often, learning comes first. Fortunately, Applitools makes it really easy for you to learn Visual AI. Just take a class on Test Automation University. There is Raja’s course: Modern Functional Test Automation through Visual AI. There is Angie Jones’s course: Automated Visual Testing: A Fast Path To Test Automation Success. And, there are others.

Learning Appium Visual Testing

Michael Battat — Thu, 09 Apr 2020 17:37:32 +0000

I’m learning Appium with visual testing in Jonathan Lipps course, Automated Visual Testing with Appium, on Test Automation University. I’m familiar with testing but new to Appium, so I figure this is a great way to get started.

Jonathan knows what he’s talking about when it comes to Appium. He serves as a lead developer and maintainer of the Appium project. His company, CloudGrey, consults on native mobile app development and testing. And, his company publishes the AppiumPro blog. I assume it’s best to learn from the master.

Learning Appium with Visual Testing – Prerequisites

Okay, Jonathan already burst my bubble in Chapter 1. He expects me to know Appium. I’m learning Appium, so I’ll do my best to be a quick study. The course uses examples in Java, but the code can be any supported language. Python, Ruby, JavaScript, etc. all work. Jonathan also wants to make sure that I understand automated testing concepts. I do.

Next, I need to have node.js installed and NPM. I need an Appium CLI installation. And, I need to have software that allows me to get Appium Android dependencies installed. Android Studio is one solution. Actually, Jonathan recommends getting all this installed before taking his course, so I figure I’ll go get a little bit more training and get the software installed. Done.

Then comes the tricky bit – installing OpenCV libraries. OpenCV installs a library of computer vision (ergo “CV”) routines to work with Appium. As Jonathan notes, there’s an easy way and a hard way to install OpenCV. The easy way uses a direct NPM command to build the libraries. He has a Mac and ran into issues with the easy way. I have a Mac and ran into the same issues. So, Jonathan provides a second way to install the OpenCV software using Homebrew.

The steps I list here are the few steps here you may run through to get Appium up and running on your local machine. As Jonathan points out, this is all documented.

On to testing.

Learning Appium Testing With A Demo App

I’m used to testing apps with web element locators. In the case of Jonathan’s demo app, it’s a game. It uses a calculator paradigm with the goal of having the user achieve a certain number by pressing on buttons that perform certain operations. Like this example:

When Jonathan tried to find traditional locators, there was only one leaf node – and it wasn’t a control button. How often does this happen? Often. Your development team can choose one of many frameworks for mobile application development that omits traditional element locators.

The app source looks like this:

You can try to hard code your tests by region geography on the screen, but what happens when you go from a small format phone to a larger phone – to an Android tablet? Yuck!

Fortunately, by using OpenCV with Appium you can define the images of buttons that you can save. Take a screen capture, crop the button of interest, and save it in a file accessible to Appium. Later you can pass that image to Appium to find a match and take an action – click. That’s really useful. To reiterate – with OpenCV, Appium can use stored images to find buttons and manipulate the app. That was new to me and really interesting.

The specific command you use is:

driver.findElementByImage(templateImage);

This command returns an ImageElement that can be used for action. The workflow between Appium, OpenCV, your test code, and a stored image looks like this:

Setting Up Your Test Device as an MJPEG Server

Part of learning Appium is learning Appium tricks. From Jonathan’s experience, Appium’s integration with the Android native screen capture and processing can be pretty slow. It can take several seconds to capture one screen. Fortunately, you can install an MJPEG server on your test device. Appium can process the MJPEG server image much more quickly.

Jonathan uses one such piece of software to connect his Android emulator to his Appium test infrastructure. Called Screen Stream, the software serves the image of the Android device so that a person or piece of software can subscribe to a URL and see what’s on the screen. Jonathan notes that the device or emulator under test must have an IP address accessible to Appium.

Once you have a stream set up, you need to give Appium access to that stream. Jonathan shows that you set the value of mjpegScreenshotUrl to your appropriate stream URL.

Example Test Code

As Jonathan mentions, his test code is written in Java, and if it’s not a language you know, his code syntax is easy to read. So, don’t let lack of Java stand in the way of learning Appium. Jonathan spends a little time showing you the code he has created for running his app tests.

His course presents the test code in easy-to-read chunks. If you’re not a Java person, you might not get all the syntax, but you’ll get why things are organized the way they are. (Note – if you’re interested in learning Java, take Angie Jones’s course about the Java language on Test Automation University).

Jonathan walks you through his test demo of an Android game that uses a calculator as the input device. And, unfortunately, there are no code hooks to drive the UI. So, Jonathan uses the image matching to drive the app. Each active button can be captured and used to drive the behavior.

You can find the source code in addition to Jonathan’s test excerpts. This will help you see the tests in more detail.

Visual Testing with OpenCV

Jonathan spends an entire chapter focused on OpenCV for visual testing. Jonathan distinguishes functional test – where the goal of testing is to find unexpected behavior in the input-output behavior of the application – versus visual test – where the goal is finding a visual behavior that may cause a user to find the app either inoperative or unappealing.

If an app allows the user to enter two numbers and add them, a functional bug results when:

Either number cannot be entered in the expected numeric fields
The numeric entry fields can receive numbers but the button click has no effect
Data entry and the button click works but the presented response does not match the expected sum.

As a tester, you can also imagine other cases for a functional test: handling negative numbers, non-standard numeric formats, and non-numeric inputs.

For the same app, an exclusively visual bug exists when:

Number fields cannot be distinguished on a page
The button cannot be distinguished on a page

Note – Appium may be able to execute a functional test even if there are visual bugs, which explains why you need a solution to examine for visual bugs.

To catch visual bugs, OpenCV lets you capture a screen from the mobile device and store it in a location on your local machine or in a network repository. Once you have captured a screenshot, that becomes your baseline for that screen. On subsequent runs, you call OpenCV to compare your screenshot to the saved baseline. OpenCV uses a pixel comparison engine, and the comparison response on the comparator comes back with a value between 0 and 1. The number is roughly analogous to the percentage of pixels unchanged.

Testing with Appium and OpenCV

To automate his tests, Jonathan sets up his Java to call OpenCV to capture a screen and compare the image to the baseline. For automation purposes, Jonathan sets 0.99 as his acceptance threshold. Anything that shows 0.99 or above generally has small differences – clock, cell signal level, etc. Anything lower likely shows a noticeable difference.

You can play with the acceptance criteria to see what works for you.

One great thing you can do is see the differences in OpenCV. OpenCV will capture a screen of both the checkpoint and the baseline and highlight the differences. You can use this comparison to determine what you want to do, either:

Declare the difference a bug and pass the information back to developers or
Declare the difference as intentional and point future tests to the most recent capture, which becomes the new baseline.

Real-World Visual Tests with OpenCV

Next, Jonathan demonstrates the visual capture with OpenCV. He goes through a demo app and captures two screen images. He saves the images as baselines. Next, he runs the app with screen modifications and captures the new images and runs his pixel comparison. Each screen that has a difference where the comparison falls below 0.99 fails his test.

Jonathan shows the process you go through using OpenCV. Running through the test the first time, there is no screen capture, so you capture your initial baseline image. The second time you run through the test, you capture screens and your tests either pass or fail. The tests end on the first fail, and you must inspect the compare image to see where the failure occurred.

In the course, Jonathan ran through his sample application. He ran a test, which included two screen captures. His code captured the screenshot as a baseline if no prior baseline existed. Each screen capture served as its own unique test. After running his test with the baseline application, he then pointed to a modified application and reran his tests using the visual comparison as the validation step. Each screenshot in the second app behaved slightly differently from the initial capture. Then we went through the results.

Reviewing the Appium OpenCV Test Results

The first test run showed one failure, because the Appium run reported the first failure as a failed test. In Jonathan’s first screen, the visual difference occurred because he added a new item to the screen. Because he expected the new behavior, he could set the most recent screen capture as the new baseline for subsequent testing.

Then, he had to rerun his test. Again, the test run stopped with an image mismatch failure. While the first image passed with the new baseline, the second image now failed. Again, comparing the images showed the highlighted difference, which turned out to be a visually unacceptable change in the location of an action button. In this case, the check image needed to be sent to developers, as this image contained a bug. This test would continue to fail until the bug got resolved.

As Jonathan shows, image comparison exposes usability changes that you may or may not desire. More importantly, you learn some of the challenges of OpenCV integration with Appium used in production testing.

First, you must manage images yourself. If you have a handful of screenshots, that could be doable. But, if you capture many screenshots, that seems problematic. Second, an image is either all-good or else partially buggy. An image with both an acceptable and unacceptable change cannot be partially accepted. So, you need to manage the partial workflow on your own. Finally, the whole notion of a percentage pixel change as acceptance criteria to account for pixels you know will change seems a little imprecise.

With this, Jonathan moves on to the next chapter – visual testing with Appium and Applitools.

Appium with Applitools for Visual Testing

Jonathan starts the next chapter helping you learn the drawbacks of using Appium with OpenCV for visual testing:

OpenCV can be challenging to install
With OpenCV, you need to manage your own baselines
Self-Managing images – you also manage compare images
Image compare errors – you will have reported failures that are not failures
No support for full-screen comparisons with scrolling screens – something we hadn’t yet covered, but you must manually scroll and capture screens and sync them up. Yuck.
You must maintain the image comparison code, which will require work from release to release.

So, funny enough, Jonathan focuses on these limitations of using Appium with OpenCV, and then he proceeds to show how using Applitools overcomes each of these.

Adding Applitools code – easy. You don’t need to manage a whole image compare library. Applitools does it for you. All you need to do is call into the Applitools Eyes SDK. After that, you need to link to your valid Applitools API Key (best to use a call to a system variable, rather than use the actual key, so that you don’t worry about publishing your key with your code) and start integrating the Applitools code into your Appium tests. Jonathan also goes through the test close steps, including the calls to Applitools to abort an open session in case, for some reason, your test logic skipped a close.

After that, it’s easy to add the Applitools Eyes calls into your test code. Instead of having to use OpenCV calls that include test compares you run and validate on your own manually, all the calls to Applitools Eyes SDK pull screenshots into Applitools. Later after your tests run, you check out the results.

Looking at Applitools Results

Here you learn the differences between running Appium with OpenCV and running Appium with Applitools.

First, your tests run through to completion. Your initial tests, like your initial tests with Appium and OpenCV, run through to completion and capture a baseline image. Unlike with OpenCV, Applitools gives you a useful UI that shows you all the captured images for that run. You don’t have to manage them on your own. When you run your comparison test, the Applitools tests run through to completion while capturing all differences. So, you don’t end up executing multiple test runs to ensure that each visual difference gets captured.

In the Applitools UI, you see all the tests that have been run and ones with differences identified as “Unresolved” but not yet classified as failures. That’s because a difference could be expected or unexpected. You either give a difference a thumbs-up – meaning that the difference was expected (and turning the checkpoint into the new baseline) or a thumbs down – meaning it was a failure.

One useful thing to note is that you can have Applitools ignore the status bar of the phone entirely and focus just on the app. So, unlike the OpenCV screen captures, which capture the full screen and requires you to do some postprocessing, the Applitools capture can select just the app in question.

Jonathan shows how the Applitools UI lets you automate the visual capture and streamline your testing. You can review capture history over time for different tests, and see what you have added or changed. It can give you a history of how your app looks over time.

Advanced Features – Learning Appium with Applitools

Now, Jonathan helps you learn some of the advanced features of Appium with Applitools. First, Applitools lets you capture the full application window – not just a screenshot. You invoke the method:

eyes.setForceFullPageScreenshot(true);

before your call to eyes.open() to start your test, and Applitools will automatically grab a full-page screenshot. Jonathan shows the difference between the prior and current tests by showing that the prior tests only looked at the first screen of the app, but Applitools could see a range of additions to the app.

Here, two different runs of the same home screen are captured. The one on the right, the checkpoint, is actually a longer screen and has more content than the baseline on the left. A full capture sometimes finds new content that gets added between test runs, as shown here. But, often, the benefit of full-screen capture for a run is the ability to auto-scroll and stitch together a single image for comparison purposes.

Next, Jonathan shows how you can use Applitools to treat different regions on a page for visual testing. Sometimes your captured screen includes content that will vary from test to test. Jonathan shows a mobile test of an echo screen, where a user types a test phrase and clicks an on-screen “OK” button. The mobile device then displays the typed text in a different region of the screen.

When you use a variable to define your test input phrase and your expected output phrase, your Appium functional test will run and, as long as the two phrases match the variable, the test will pass. However, you will have a visual difference if you change the phrase while capturing the image with Applitools. So, Jonathan shows you how to treat a region as “ignored”. You can select a region, identify it as ignorable, and then Applitools ignores the regions.

Jonathan’s exception demo shows some of the power in Applitools, but the use case might be better for a region that changes independent of the test being run. For instance, if the test screen includes a digital clock region that shows the device time, then that clock will always show a different time. When you run your tests, you might want to ignore the clock region as its region will always change and will always give you an error. You likely want to run and capture multiple versions of the input-output test to handle things like text wrapping and scrolling.

Conclusions

Wow. Appium provides powerful tools for testing mobile apps in both Android and iOS. While Jonathan’s course focused on Android, and runs his tests on an Android emulator, I can easily imagine doing similar tests using iOS.

By focusing on apps that lack traditional locators, Jonathan shows how mobile device testing can depend on visual capabilities for both applying tests as well as measuring results. And, he shows both the benefits of using a package like OpenCV to add visual capabilities for both applying tests and measuring results, and he also shows some of the limitations of OpenCV in production visual validation. Those included the challenges of managing images manually as well as having the first visual failure with OpenCV causing a test run to fail and not collecting all the visual differences for a test run.

Finally, Jonathan does a great job showing how Applitools overcomes a lot of those OpenCV limitations and actually provides a valuable tool for production test automation.

How do you bridge the gap between quality engineering and product management?

Michael Battat — Thu, 02 Apr 2020 17:41:34 +0000

In October 2019, Evan Wiley from Capital One presented on this topic for an Applitools webinar. Evan introduced a number of cool tools that bridge the gap between the product specified by the product managers and the nuts and bolts of testing.

Evan Wiley spent six years in Quality Engineering at Capital One before moving into product management. In his engineering time, Evan discovered that product managers and quality engineers share complementary views on their products. Product managers envision behaviors that quality engineers execute in their tests.

Evan experienced this relationship directly when he was invited by product managers to attend “empathy interviews.” In these interviews, team members speak with customers to understand the customer’s environment, needs, fears, and expectations. In attending these interviews, Evan heard the first-hand lives of customers who were using the results of his work. These interviews both informed his work in quality engineering and later fueled his move to product management

What Is Quality Engineering?

Evan described the work of quality engineering as finding bugs within products early and often. Noting that the job varies from organization to organization and company to company, Evan described the responsibilities including:

Manual testing – ensuring that the product can be exercised and behaves as expected
Test automation – creating automated tests that can be run to assure behavioral coverage without manual intervention
Production testing – verifying behavior with a production-ready product.
Test case design – ensuring that tests cover both positive cases, or normal function, as well as negative cases, or handling problematic inputs, internal inconsistencies, and other errors.
Test execution – running tests and reporting results
Penetration testing – running tests and reporting results for tests based on known threat vectors
Accessibility testing – ensuring that customers with known visual and other disabilities can navigate and use the product successfully.

Evan noted that the nature of this work changes as products, product delivery platforms, and customer environments evolve. And, things change constantly.

What is Product Management?

Evan next dove into describing the role of product management. Frankly, describing product management can take a two-day course, and even then not cover what everyone thinks product management might do at their company. I know this as I remain a poorly-hidden closet product manager.

Evan does not try to describe product management comprehensively. Instead, he focuses on how empathy interviews drive product managers to make product decisions. Primarily, Evan says, empathy interviews help product managers become the “voice of the customer.”

One part of empathy interviews guides testing. For example, does your test environment match the customer’s environment? Do your test conditions match the customer’s test conditions?

A larger set of questions helps product managers understand problems their customers try to solve, how to prioritize solutions to these problems, which of these problems need to be higher on the near-term backlog versus further back, and how customers might respond to different marketing messages.

And, when product managers take their products to the field, they can validate a customer’s reaction with expectations from empathy interviews to help make their empathy interviews even more effective. The initial empathy interview forms the basis for product management key performance indicators.

Ultimately, the needs of product managers and quality engineers diverge significantly. But involving quality engineering in customer empathy interviews help the overlap succeed.

Quality Engineering and Product Manager Needs Overlap

Quality over Quantity

Evan spends a bit of time discussing how Capital One prioritizes quality over quantity. Evan points out that the company makes this choice, and that the decision permeates the company culture. At Capital One, that goal permeates all engineering – not just quality engineering.

Evan explains with an example from another company.

“If there’s a quality engineer at Facebook, they might have a lot of test cases, and in some cases, they can stand in for the end-user with the knowledge of being a user. So, if they’re testing, say, logout from Facebook, they might think, ‘I can make that simpler for an end-user because I’d want it to be.’ And this insight empowers the quality engineers to work directly with the developers tweak a behavior for an end-user.”

To this end, Evan sees that the whole of engineering contributes to quality over quantity culture. Hiring processes, skill selection, and testing approaches involve transparency that allows for a breadth of experience and diversity of perspectives on the team.

And, this mix of backgrounds leads to cross-training product managers with quality engineers. Bringing both groups together inside Capital One leads, in Evan’s perspective, to better outcomes for customers.

Evan gave the example of testing a set of features across multiple browsers alongside product managers. He was able to show the product managers where the different browsers handled certain functions differently as a way to build the culture of quality at Capital One.

Gherkin Scenarios

Next, Evan demonstrated the use of Gherkin Scenarios for writing stories that describe the behavior of a product. If you don’t know about Gherkin, it’s basic statements are:

Scenario
Given
When
Then
And

So, for example, Evan talks about the Google home page. He imagines the product manager writes something like this:

These scenarios have several useful properties.

First, they help the product manager describe the detailed behavior a user will experience before any software gets written. Product managers can validate these stories with prospective customers and identify any issues that might impact the behavior of the product.

Second, developers get an understanding of the intended user experience and outcome to plan their coding activity.

Third, engineers can use their experience to determine scenarios that might not have been described and what should be the intended behavior in those situations. And, engineers can ask relevant questions involving both design as well as behavior.

A Gherkin Example

Imagine some questions that come from the scenario listed above. Here are a few:

How much time should the user experience between clicking the “Google Search” button and getting a response back?
What happens when the user has 600ms latency between the browser and server?
Since the scenario specifies that the test uses valid text for a Google search, define valid text.
What about the scenario when the user enters not valid text?

These questions lead to more scenarios, which lead to more complete product specifications, which lead to more complete products. The larger number of scenarios also leads to more complete testing. Quality engineers read these scenarios and begin planning their acceptance tests based on these different scenarios and conditions.

So, much of the engineering-product management conversation ends up as quality engineering talking with product management about scenarios – tightening the connection between product management and quality engineering.

Evan did not talk about it, but a tool like Cucumber reads in Gherkin scenarios to help build test automation.

Visual Validation Baselines

From there, Evan moved on to talk about visual validation. And, for visual validation, Evan talked about Applitools – a tool he uses at Capital One.

First, Evan spoke about the idea of user acceptance testing. When you run through the scenarios with product managers, you will end up with the screens that were described. You want to capture the screen images and go through them with product managers to make sure they meet the needs of the customers as understood by the product management team.

So, part of the testing involves capturing images, and that means following your Gherkin scenarios to make sure you capture all the right steps. Evan showed some examples based on a Google page and describing how those test steps in the Gherkin scenarios became captured visual images. Evan pointed out that these visual images begin to define how the application should behave – a baseline for the application.

As you go through the different pages, you can complete the baseline acceptance tests. Once you have a saved good baseline, you know the state of an accepted page going forward.

If you find a behavior you don’t like, you can highlight the behavior and share it with the developers.

You can find problems during visual testing that do not appear in development. For instance, someone realizes you need to test against some smaller viewport sizes. Or, you test a mobile app, but not the mobile browser version.

So, you build your library of baseline images for each test condition. You make sure to include behaviors on target browsers, operating systems, and viewport sizes in your Gherkin scenarios. As it turns out, with Applitools, your collection of completed baselines makes your life much easier going forward.

Visual Validation Checkpoints

Next, Evan dove into using Applitools for subsequent tests.

Once you develop your test cases and run them through Applitools, you have baselines. As you continue to develop your product, you continue to run tests through Applitools. Each test you run and capture in Applitools can be compared with your earlier tests. A run with no differences shows as a pass. A run with differences shows up as “unresolved.”

Evan showed how to deal with an unresolved difference. He inspected one difference, saw it was due to an expected change, and accepted the difference by clicking the “thumbs-up” button in the UI. In this case, the checkpoint automatically becomes the new baseline image. He inspected another difference. In this case, the difference wasn’t one he wanted. He clicked the “thumbs-down” button in the UI. He showed how this information becomes a “failed” test, and how the information of the can get routed back to developers.

Unlike other visual testing tools you have used, the Applitools uses computer vision to identify visual differences. The Visual AI engine can ignore rendering changes at the pixel level that do not result in user-identifiable visible differences. And, there are other capabilities for maintaining applications, such as handling changes that displace the rest of the content on pages or managing identical changes that occur on multiple pages.

Quality over Velocity

Evan went back to discuss the company culture about prioritizing quality. Capital One developed an engineering culture over time to focus on quality. Any decision to emphasize delivery over quality must be documented and validated. Release decisions at Capital One end up being team decisions, as the whole team is responsible for both the content and quality of a release. So, the entire decision to deliver quality products brings the product management, product development, and quality engineering teams together with a common purpose.

Evan noted that, in his experience, other companies approach these problems in different ways. The culture at Capital One makes this kind of prioritization possible. Cross-training makes this delivery possible because cross-training makes all team members aware of priorities, approaches, and tools used to deliver quality. The result, says Evan, is a high-performing team and consistency of meeting customer expectations.

Questions about Quality Engineering

A questioner asked Evan if Quality Engineering at Capital One had sole responsibility for quality. Evan said no. Evan pointed out that he spoke from his perspective, and while Quality Engineering came up with the approach to validate product quality, the whole team – product management, development, and quality engineer – participated in the testing. The approach helped the team deliver a higher quality product.

Another questioner asked about the benefit of getting customer knowledge directly to Quality Engineering. That’s valuable, Evan said. For example, during an empathy interview, a customer raises the question of a specific problem they have when trying to execute something specific. During the interviewer, the interviewer dives deeper into this issue. The result is a more complete understanding of the customer use case, the expected behavior, and the actual behavior observed. This results in both better test cases as well as future enhancements.

Questions about Visual Testing and Tools

A questioner asked if Gherkin scenarios made sense in all situations. Not always, said Evan. Gherkin scenarios make great sense when fitting into behavior for development to create and quality engineering to test. Evan thought about cases, such as technical debt cases, for which the intended behavior may not be user behavior.

Another questioner asked about the value of visual testing to Capital One. Evan talked about finding ways to exercise a behavior, capture the results, and share the results with product people. Test pass/fail results could not capture the user experience, but visual test results do so as part of the visual testing solution. One example Evan gave was for a web app that had an unexpected break on a mobile browser, due to the different browser behavior on a different operating system. Without visual testing, the error condition would likely not have been caught in-house. If Capital One were only using manual tests, the condition might not have been covered if the specific device version was not included in the test conditions. With the automated visual tests, they found the problem, saved the image, and used that as a new Gherkin scenario in the next release.

Questions about Product Management and Quality Engineering

Next, Evan was asked about how to integrate product management and quality engineering more closely. Evan said he wasn’t sure how to do this in the general case. At Capital One, the need for engineers and product managers to collaborate on issue grooming, with the ability to capture the visual behavior during a test run, improved the ability of the engineers and product managers to agree on issues that needed to be addressed, in what priority, and for what purpose.

Finally, Evan was asked how to get Product Management to involve engineering more closely. Evan focused on empathy interviews as ways to align engineering and product management, and Gherkin scenarios as tools to bring a common language for both development and test requirements. Evan also talked about his own transition from Quality Engineer to Product Manager – and how he went from being tool-and-outcome focused to customer-and-strategy focused.

Whole Team Testing for Continuous Delivery

Michael Battat — Thu, 26 Mar 2020 15:16:58 +0000

I just completed taking Elisabeth Hocke’s course, The Whole Team Approach to Continuous Testing, on Test Automation University. Wow! Talk about a mind-blowing experience.

Mind-blowing? Well, yes. Earlier in my career, I studied lean manufacturing best practices. In a factory, lean manufacturing focuses on reducing waste and increasing factory productivity. Elisabeth (who goes by ‘Lisi’) explains how this concept makes sense in a continuous delivery model for software.

The Lean Factory

To learn lean manufacturing, I read a book by Eliyahu Goldratt called The Goal. Written in 1984, the book helped explain the concept of waste as building up an inventory of work-in-process (WIP) that could not be manufactured. Goldratt describes walking into a factory filled with WIP inventory that was just piling up.

As Goldratt describes the problem and wonders how to address it, he goes on a hike with his son’s boy scout troop. One member is always the slowest. When they put that scout at the end, he falls behind everyone else. When they put that scout in the middle, it becomes two groups – the fast group, then a gap, and then this scout and everyone else behind him. Finally, they put this one scout in the front of the troop, and everyone stays together.

Goldratt describes the ‘aha!’ moment of realizing that the slowest process in the factory limits the speed of the factory. Everyone else might be busy building body parts or electronic harnesses, but if some step further in the assembly process can’t consume those parts as quickly as they are made, they build up waste.

Goldratt concluded that the efficient factory designs its process to make products at the pace of the slowest process – that profitable operation creates finished products and not waste.

Continuous Delivery As Lean Factory

Lisi spends the first chapter delving into the definition of continuous testing in the framework of continuous delivery. She quotes industry thought leaders like Bas Dijkstra, Jez Humble, Dave Farley, and Dan Ashby to get to some key ideas – namely, that continuous testing means ensuring that you are testing at every step in the continuous delivery process. She even shares this image from Dan Ashby:

This diagram shows continuous delivery in a DevOps model with testing everywhere.

Lisi makes the key point – success in continuous delivery means shortening feedback loops to learn early. Every point of development and delivery needs validation.

In planning – how do you test your plans? How do you validate what you’re up to?
In branching – how do you make the choice about branch design?
During coding – well, is it more than unit tests? And who runs the tests?
In the merge – what are you testing?
Etc.

She focuses on the key idea that at each step you make choices and want to know if those choices work. Every choice has consequences. Which designs work? Which design choice provides you with optimal results? How do you validate the performance impacts of a certain design choice? All of these questions result in choices you can test.

If you are a test professional and not in this system, you must be wondering what this means for your role. As in – what kind of testing can you do without code to validate? Or, for a developer, how can anyone test code before code completion?

Lisi walks through examples that turn you from coder or tester to scientist. Everything you do becomes a test. You can run large tests or small tests – but test early and learn to get test results early.

Unproductive Work As Waste

I brought up the lean factory because Lisa raises a lean manufacturing concept in her description of work in a continuous integration project. Lisi describes waste as the enemy of continuous deployment.

Any work that fails to deliver value involves waste. In a factory, work that creates unfinished parts faster than the delivery of finished goods creates waste. Usually, you see waste as piled up inventory. In software development, your processes can also create waste. Usually, though, the waste comes in the form of code in a “dev-complete” state – simply waiting to be tested. But, there are other sources of waste – uncoded designs, and unmerged code that are work that cannot be turned into the next step

And, with that, we come full circle to the application team. Does your team divide the coding responsibility from the verification skill set? Do you create mini waterfalls? Or, do you build a team that tries to do something different – to reduce waste in processes, to deliver quality in software, and to deliver products more quickly to market at the pace of the team?

Whole Team Testing For Agile Software Delivery

Here, Lisi turns to the Whole Team testing approach, which becomes a way of thinking about making the entire team responsible for delivering a finished product.

Lisi uses this quote from Lisa Crispin and Janet Gregory from their 2009 book, Agile Testing: A Practical Guide for Testers and Agile Teams:

“…everyone involved with delivering software is responsible for delivering high-quality software.”

Lisi gives examples from her own career to back up this idea. Her own team had gotten bogged down in undelivered code, error tickets, and general frustration. They tried the lean approach for software – “stop starting and start finishing.” Everyone pitched in to help – no matter what role they held. People helped code, test write automation. People helped the product owner – or another product owner. The team worked to finish what they had started.

As a result, Lisi’s team built up trust and broke down barriers. While expertise might exist in pockets, they realized that delivering software required knowledge sharing and growing as a team. Testers know how to test software – but they cannot be the only people writing tests if the team hopes to deliver effectively. Lisi explained that, once the team had collaborated together once, they were prepared to do it again.

So, Lisi asks us to think about our own teams and how we build and share knowledge. We might have silos. We might only have a single test automation expert. She contrasts that with her teams, which work to share knowledge and focus on getting things done.

Her course offers lots of resources to help you move to a more team-oriented approach. One of those is the set of courses on Test Automation University, which can help you increase your team’s skillset in automated testing.

Organizing Your Team

Lisi spends three chapters talking about organizing your team for success.

Working Solo

First, she looks at organizations where everyone ‘works solo.’ Individuals do their own tasks and try to be as productive as possible. Tremendous amounts of specialization. Plenty of opportunities for waste. Why? Because individuals measure their productivity based on delivery relative to personal productivity goals.

In the world of the individual practitioner, everyone works at different rates. A developer finishes unit of work ‘A’, hands ‘A’ to a tester, and begins working on unit ‘B’. The tester gets testing ‘A’ and finds a bug. When the tester hands it back to the developer, the developer has to context switch to ‘A’ to revisit the code and remember why the bug occurred to fix it. Working Solo involves lots of context switching – which introduces waste.

Another common source of waste comes from team imbalance. For example, your team has sufficient numbers of developers to build an app, but you haven’t hired enough testers to ensure that the app behaves as expected on all user platforms. If you constrain individuals in their silos, your team may falsely see itself forced to choose tradeoffs that increase the likelihood that you will release untested code.

If your team runs solo and you reward individuals for their personal productivity, you might be surprised at how the team doesn’t meet all the schedules that matter to you.

Pairing

Next, she looks at ‘pairing.’ Engineers work together as pairs. They can be colocated or work remotely, but they work together. In some cases, they share ideas and divide tasks. In the more explicit cases, no member of the team can both come up with ideas and write code. Someone with an idea has to explain it to the other, who actually writes the code. In pairing, the team shares ideas and experience – and learn together from each other.

Pairing helps the overall team write code that incorporates more robust and thoughtful design as two minds work together to deliver pieces. And, two people who work together learn from each other. Successful airing builds trust and team collaboration.

Mobbing

Finally, Lisi looks at ‘mobbing.’ In mobbing, a large group of people comes together to think and create. Mobbing, one person takes the keyboard and acts as the ‘driver’. The rest of the people suggest ideas and must explain to the driver what they mean to build what they are explaining.

A mob with a lot of ideas can seem chaotic – and it can lead to lots of teamwork… if you are deliberate in mobbing and use some thoughtful rules, your mob can be effective. The most important parts:

Keep everyone engaged – be in a place where you are either contributing or learning from others
Use the “yes, and…” approach from improvisational theater
When you have multiple ideas, try both – starting with the person with least experience (remember, it’s about learning)
In the mob, everyone is learning – be kind and respectful

When To Use Which Approach

Finally, Lisi goes back to the idea that there is no ‘one size fits all’ to collaboration styles. In some cases, teams can see that certain problems can be divided into solo work. In some, they will gravitate to pairs. And, at times, they will call for a mob.

Her point, though, is that teams must know these approaches exist and have experience with all to find which will be the best for the team output in default as well as knowing how to handle the others when necessary.

The key metrics involve team productivity – not just individual productivity. If you track individual productivity without tracking team waste, you don’t know where you are building inefficiencies. And, once you begin to track team productivity, you start to see which approaches work best for you as a development organization.

One hidden productivity benefit of collaboration comes from interdependence. If only one person understands customer needs, or specific test approaches, workflows depend on that person always being part of the loop. While that may seem to give those people special power of knowledge, they also seem personally constrained so they cannot take a vacation or even a sick day without their absence impacting team productivity.

When individuals share their knowledge, two really good things happen. As teams start tracking the flow of work through the team, they see that the expert can take time off without impacting team productivity. And, for the experts, they can continue to develop their knowledge in different areas so they don’t become a siloed specialist whose quest for new knowledge and expertise becomes blocked by solving the same problem all the time.

Whole Team Testing – Conclusions

Before I get started with my other conclusions, I want to mention amazing resources. Lisi includes a generous set of links to both paid and free resources you can use to learn more and share with your team. The course serves as a starting point to what could lead to a major change in the way you work.

As I took the course, I realized that ‘Whole Team” means more than just engineers. Lisi made it clear that everyone in the product delivery cycle plays a role. As team members collaborate and learn, the team can become more productive. To succeed, the team needs a willingness to experiment and tools to measure effectiveness.

Lisi got me to realize that the factory view of waste, which I read about in The Goal, makes sense in a software context as well. Creating software means creating value. The value for the software maker starts with the value that customers obtain – which means building high-value, defect-free experiences for customers as they use the product.

Similarly, I had not thought about experimentation in ways to get faster team delivery, and that group collaboration styles could impact team productivity. I have worked on projects that tracked team velocity. I could easily see adding this approach to teams in the future – and not just in software development.

Lisi’s point wasn’t simply that collaboration matters during coding. I inferred that we should look for opportunities to use pairing or mobbing – even during design, planning, or release. Her point, as I saw it, was that a rigid approach to team structure limits team productivity. By adding collaboration and work style skills to a team, we increase the opportunities for the team to increase team velocity and productivity.

The Caveats of Modern Functional Testing

Michael Battat — Wed, 18 Mar 2020 22:14:04 +0000

In reaching the conclusion of the course on Modern Functional Testing using Visual AI, I reach the key question:

Can I use visual assertions to replace text-based assertions for my test infrastructure?

In my prior blog posts, I have summarized each chapter of Raja Rao DV’s course on Modern Functional Testing, which you can take on Test Automation University. We have reached Chapter 8 – Three Main Caveats. Both this chapter and the prior chapter help frame the real-world issues involved in modern functional testing with Visual AI. And, they help frame the answer to the question you’re undoubtedly asking yourself – will Visual AI testing work for me?

Let’s discuss the caveats and then draw some conclusions

Caveat 1 – Dynamic Data

We already covered dynamic data in Chapter 7, which explained how Applitools Visual AI incorporates region-based exception handling for truly dynamic parts of a web page.

If all you had was a full screen or full page snapshot, dynamic data would generate differences randomly in those dynamic regions. You could not let the app run freely – you would have to constrain its behavior.

However, if you can apply different match levels to different regions, you can handle dynamic parts of a page. In one case, you can simply ignore a region. Or, you can ensure that the region layout remains structured correctly – even if the content changes.

So, handling dynamic data is one caveat you already know you can overcome.

Caveat 2 – Browser Limitations

The second caveat involves specific browser or app limitations. Raja gives the example of browser-native capabilities, such as native select menus or native pop-up windows. Neither of these coding structures lends itself to snapshot capture. If you cannot capture the visible behavior, you cannot run a screen comparison.

If you use these kinds of structures in your app, visual AI cannot capture the behavior. You may still need to use legacy assertions to capture this behavior.

Caveat 3 – Test Execution Speed

The third caveat involves test execution speed. It takes time for a browser to render a page. On top of rendering time, visual testing may be instructed to scroll and capture an entire page – which is done a screen at a time. Scroll-and-capture can cause the entire test run for an app to take considerable time.

Yes, the legacy approach can seem faster. Using code to inspect elements in the DOM runs much more quickly than scrolling and capturing. But, as Raja has pointed out, the legacy approach becomes a management headache when developers add new features or change design that result in visible errors.

To address this caveat, Raja talks about smart test design. As the test engineer, you know the test states for your app. If any of your tests encounter a page in an identical state, capture that page only once. Also, to reduce execution time, run your tests in parallel as much as possible.

Revisiting The Course

To sum up my series, I thought I would revisit some of the takeaways that Raja shared or that I inferred through his course.

Less Code To Manage

Compared with legacy functional tests, a modern functional test with Visual AI leaves you with less code to manage. And, this matters over time, because while developers can make small changes, even small changes can have a large impact on the code you have to manage.

When you develop functional tests with code assertions, you trust that your code assertions will catch a programming error that causes the expected code not to be where you expect it. You figure that locators will guide you to the proper line of HTML and that the existence of the value related to that locator describes the screen behavior.

Except, of course, it is possible to have locators remain the same and cause an app to become unusable. For example, your developers can introduce a small CSS change. That change does not modify your locators, but it does make your web app unusable. Would your existing legacy code expose the impact of overlapping, hidden, or invisible content?

By reducing dozens of code assertions on a single page to a single visual assertion, you reduce the amount of code you have to manage. Visual assertions will catch your visual failure modes as well as your functional failure modes. And, if developers fail to tell you about a change or new features, your visual assertions will expose those changes (and, maybe, help improve your team communications).

Covering More Cases

The next benefit we discussed involved problematic test cases for legacy functional test. These included:

Testing tables, where table data can change in large chunks through ordering and filtering. You can code your tests by following the code logic – and discover that your test instructions can validate code that, in fact, does not match user expectations.
Running data-driven testing, where it becomes too easy to code only for expected behavior and miss unintended or uncommunicated changes.
Handling dynamic content, which can change from test run to test run or app instance to app instance. In such cases, you either don’t bother testing, or you write code to test sections and structure of the page and ignore visual representation completely
Testing complex HTML structures – like iFrames. These might be difficult to navigate for web element locators. Your code might be unwieldy to maintain over the long term.

For these kinds of test cases, text-based assertions are either difficult to code, difficult to recode when changes occur, or impossible to code in the first place. If you use these types of structures in your code, you are less likely to adopt functional or layout changes – as those changes introduce a test code maintenance cost in addition to the direct coding cost.

Visual assertions handle all these cases. Tables become a single assertion instead of a set of coded assertions. Data-driven testing loses its code dependency and can catch unintended behavior or visual changes. Dynamic content you can contain and step through to treat as static or snapshot content – or you can treat those regions as ignore regions. And, finally, snapshots can handle more complex HTML code.

Faster To Develop, Easier to Maintain

With less coding and less code required, visual testing with visual AI simplifies your test development process. Exercise a test – eyes.checkWindow(). Exercise another test step – eyes.CheckWindow(). You begin to manage where you capture screens – and that management step becomes the skill set you care about – rather than the code management for tests of all the visual elements you want to verify (and finding ones you missed).

Once you have baseline images, each subsequent run becomes a new checkpoint. All identical images pass. Any changes between the checkpoint and baseline are called “Unverified.” You get to inspect those differences and either accept or reject the changes. You can see the differences on-screen – either side-by-side or overlapping. That makes it easy to determine which version you are seeing, and whether or not you want to approve or reject the change.

We spent some time reviewing the Automated Maintenance feature. Automated maintenance lets you maintain tests by finding a change on a single screen, accepting or rejecting it, and then using Automated Maintenance to perform the same accept or reject on all identical changes found within your test set. If you introduce a new menu item to all pages of your app, Automated Maintenance lets you handle that change quickly.

Exception Handling

We spent Chapter 7 reviewing how to handle dynamic data and other exceptions. You saw how you can easily capture regions and treat them differently from the rest of the capture. You can ignore regions, just validate the layout, or just validate the content. Be careful with exceptions – they will persist as you update the app, and may not always be relevant for an upgraded app.

Conclusions

I started taking Raja’s course as a way to figure out what you would experience if you were to take the course. As an Applitools team member, I know more than a novice – but I don’t have all the experience that some test engineers bring to application testing.

We discussed internally what we thought you might take away from this course – and what might hold you back from trying Visual AI. We recognized that anyone who thinks of functional testing as entirely code-based entry, execution, and assertion, would think of visual testing as a nice-to-have add-on to their functional tests. But, the more we thought about the limitations of functional testing with code assertions, we realized we had identified the problem plaguing application testing.

In my subsequent conversations with QA practitioners, I realized that everyone has built or used some kind of snapshot tool in their QA process for years. As one of my former colleagues noted, he used snapshot comparison at our prior employer to figure out if the developers had introduced new features without telling QA.

However, no one has used visual snapshots in production and test automation, because no one could automate the visual testing process with any degree of accuracy. Automated tools reported too many false positives. Visual testing indicated potential issues but could not be relied upon in production test automation. So, visual testing could only be a nice-to-have.

We think we have built that production-ready test automation tool that increases your productivity as a tester. I hope you have gotten interested enough in Applitools to add Visual AI to your functional testing process. Depending on your test language and test infrastructure of choice, you can find a tutorial on applitools.com and even a course on visual testing in Test Automation University.

How to use Visual Validation Testing with Dynamic Data

Michael Battat — Thu, 12 Mar 2020 18:34:43 +0000

You’re running functional tests with visual validation, and you have dynamic data. Dynamic data looks different every time you inspect it. How do you do functional testing with visual validation, when your data changes all the time?

I arrived at Chapter 7 of Raja Rao DV’s course on Test Automation University, Modern Functional Test Automation Through Visual AI. Chapter 7 discusses dynamic data – content that changes every time you run a test.

Dynamic data pervades client screens. Just look at the screenshot above. Digital clocks. Wireless signal strength. Location services. Alarm settings. Bluetooth connectivity. All these elements change the on-screen pixels, but don’t reflect a change in functional behavior.

This bank website updates the time until the local branch closes, in case you want to visit the branch. That time information will change on the screen. And Visual AI captures visual differences. So, how do you automate tests that will contain an obvious visual difference?

Dynamic regions can be the rule, rather than the exception, in web apps. But, for the purposes of visual validation, dynamic elements on-screen comprise the group of test exceptions. You need a way to handle these exceptions.

Match Levels

Visual validation depends on having a good range of comparison methods because dynamic data can impact visual validation. Visual AI groups pixels into visual elements and compares elements to each other. The strictness you use in your comparison is called the match level.

Applitools Visual AI determines the element relative location, boundary, and properties (colors, contents, etc.) of each visual element. If there is no prior baseline, these elements are saved as the baseline. Once a baseline exists, Applitools will check the checkpoint image against the baseline.

Raja introduces three kinds of match levels for Applitools to compare your checkpoint against your baseline. You can use these match levels to inspect a subset of a page, a screenful, or an entire web page. Here are the three main match levels:

“Strict” – Visual AI distinguishes location, dimension, color, and content differences as a human viewer would.
“Content” – Visual AI distinguishes location, dimension, and content differences. Color differences are ignored as long as the content can be distinguished. Imagine wanting to see the impact of a global CSS color change.
“Layout” – Visual AI distinguishes location and dimension and ignores content like text and pictures. This match level makes it easy to validate the layout of shopping and publication sites with a consistent layout and dynamic content.

You choose the match level appropriate for your page. If you have a page with lots of changing content, you choose “Layout” – which checks the existence and of regions but ignores their content. If you just made a global color change, you use “Content.” In most cases, you use “Strict.”

You set the match level in your call to open Applitools Eyes. If you don’t specify a match level, it defaults to “Strict.”

Handling Exceptions – Regions

We like to think of our applications on a page-by-page basis. Each locator points to a unique page that behaves a certain way. In some cases, the relevant application content resides on a single screen. Often, though, applications and pages can extend beyond the bottom of the visible screen. Occasionally, content extends across wider than the visible screen as well.

By default, Applitools captures a screenful – the current viewport. Raja covered this code specifically in the prior chapters when he showed us how to use:

eyes.checkWindow();

Using the same command, with the “fully” option, you can capture the full page, not just the current viewport. Assuming the page scrolls beyond the visible screen, you can have Applitools scroll down and across all the screens and stitch together a full page. So you can compare full pages of your application, even if it takes several screens to capture the full page.

Be aware that the default comparison uses strict mode. You can choose a different mode for your comparison. And, you can handle exceptions with regions.

So, now that you know that you can instruct Applitools to capture a full page, or a viewport, what happens when you have dynamic data, or other parts of a page that could change? You need to identify a region that behaves differently.

Applitools adds the concept of “regions.” As Raja describes, “region” describes a rectangular subset of the screen capture – identified by a starting point X pixels across and Y pixels down relative to the top of the page, plus W pixels wide and H pixels high.

Control Region Comparisons

Once you have a region, you can use one of the following selections to control the inspection of that region:

Ignore – ignore that region completely. Its contents do not matter to the test identifying differences. Useful for counters
Floating – the content within the region can shift around. Text that can shift around.
Strict – content that should stay in the same place and color from screen to screen
Content – Content that should stay in the same place with varying color from screen to screen
Layout – Content that can change but has a common layout structure from screen to screen

Regions let you be permissive when you use a restrictive match level. “Ignore” literally means that – ignore the content of the region. There may be times you want to ignore a region. More often, though, you might want to ensure that the region boundary and content exists – for this you use “Layout.”

Regions let you handle exceptions on a more restrictive basis as well. For example, on a page using layout mode, you can create a region and use “strict” to compare content and color that should be identical – such as header or menu bar.

Testers Choice

One big point Raja makes is that you get to choose how to deploy Visual AI. Select the mode that matches the page behavior you expect, and then set the appropriate mode for handling exceptions.

Raja demonstrates how you can choose to define exceptions in the UI or in your test code. You can choose to set the exceptions in the Applitools UI. Once you set a region with a specific match level, that region with that match level persists through future comparisons. Alternatively, you can add regions to handle exceptions directly in your test code. Those region definitions persist as long as they persist in your code.

You don’t need to capture an entire window or page. You can run your eyes.open() and use:

eyes.checkRegion()

Choose just to capture individual regions at an appropriate comparison level. This kind of test can be useful during app development when you want to distinguish the behavior of specific elements you are building.

If you’re really focused on using element-based checks, you can even run:

eyes.checkElement()

The checkElement instruction uses web element locators to find specific on-page elements. checkElement lets you use a legacy identifier in a modern functional test approach. In general, though, checkElement adds more complexity compared with visual validation.

The key understanding is that, for a given capture page, you can define your mode for the full capture and exceptions for specific regions, so that you cover the entire page.

Handling Expected Changes

When you make changes, all your captures must be updated. CSS, icons, menus, and other changes can affect multiple pages – or even your entire site. Imagine having to maintain all those changes – page by page. Yikes.

Fortunately, Applitools makes it easy to accept common changes across multiple pages.

Whenever you encounter a difference on a single page, you are instructed to accept the change or reject it. If you reject the change – it’s an error and you can flag development. But, if you accept the change, you can also use a feature called automated maintenance to accept the change on all other pages where the change has been discovered.

Update your corporate logo. Done.

Install a new menu system. Easy.

You can use Automated Maintenance to accept changes. You can also use Automated Maintenance to deploy regions across all the pages – such as ignore regions.

Of course, the more comprehensive your changes, the more challenging it is to use automated maintenance. If you make some significant changes in your layout, expect to create new baselines as well as use automated maintenance.

Conclusions about Visual Validation and Dynamic Data

We all want to build applications that achieve business outcomes. We often build visually interesting pages with changing content designed to keep buyers engaged. But, we also know that testing requires repeatability – meaning that dynamic may be great for business, but testing requires predictable results.

Dynamic data can limit the benefits of visual validation. You need a way to handle dynamic data in your visual validation solution. Applitools gives you tools to handle dynamic parts of your application. You can handle truly dynamic sections by ignoring regions, treating those regions as layout regions, or even handling a whole page as a layout and let sections and content change.

And, when you make global changes, automated maintenance eases the pain of updating all your baseline images.

As Raja makes it clear, Applitools has thought not just about discovering visual changes, but handling unexpected changes that are defects, dynamic data that will produce false-positive defects, and expected global changes affecting multiple pages. All of these features make up key parts of a modern functional testing system.

How To Simplify Complex Functional Testing

Michael Battat — Tue, 03 Mar 2020 18:53:24 +0000

How does functional testing with visual assertions help simplify test development for complex real-world apps? Like, say, a retail app with inventory, product details, rotating displays, and shopping carts?

My special blog series discusses Modern Functional Testing with Visual AI, Raja Rao’s course on Test Automation University. I arrived at Chapter 6 – E-Commerce Real World Example. In this review, I hope to give you an overview of Raja’s examples and how they might apply to your test challenges.

Real-World Challenges

Raja starts by explaining the challenges in creating sophisticated functional tests for an e-commerce app. His demonstration site lets a shopper:

Scroll inventory
Select items
Put items in a shopping cart
Delete items from a cart
Process transactions

The app includes featured items that can change each time a shopper visits the site.

In this case, Raja uses an e-commerce app from SAP, with sample data and a sample setup. The SAP application includes all the features of any e-commerce site, and it does have a simple look-and-feel for all that complex functionality. It’s useful for demonstrating the challenges of implementing a complicated test for the app pages and functionality, which I’ll dive into in a moment. But first…

There may be those among you readers who think,

“Hey, this isn’t a legitimate real-world test. He’s using a canned application. How does that compare with complex real-world apps we code and test ourselves?”

I have to admit that I was thinking that for a sec. And then I remembered all the times that people hesitate in applying app upgrades – even when there’s a security issue. Why? Because nobody feels comfortable about how that app will behave in the areas outside the issue that got ‘fixed.’ How do we know how an app we have customized will look and behave after the upgrade? How many of us have felt burned once we discovered that the vendor’s testing likely didn’t include our specific configuration?

Hold that thought – I’ll get back to it in a sec.

Challenges Testing Complex Applications

In the first few chapters, Raja focused on individual technologies that result in complex functional tests. He covered testing tables, data-driven testing, testing dynamic content, and testing iFrames. With each of these technologies, a small change on a single input can have a significant change on the output. For example, how do you ensure that the sort function on a table behaves as expected?

All of these tests share a common test strategy: perform an action, and then assert that the output matches expectations. Legacy functional tests require the test coder to assert each expected element exists in the DOM. Testing gets complicated by many conditions – but let’s boil them down to three:

A small change in an input makes a big change on the output that needs to be checked
An app gets updated – and locators and formatting change
Lots of locators exist on the app page

And, in each of the prior chapters, Raja shows how visual assertions with Visual AI result in functional testing simplified.

Visual vs Coded Assertions

The common issues involve all the coded assertions of outputs. At a root level, do you inspect the entire DOM? Or, do you just inspect the elements you expect to change?

Some testers get functional test myopia – they focus only on the elements and behavior they expect to change in their tests. These testers think that checking every element on every page after every change seems silly. They make a change and look for expected behavior.

When you test a table sort or some other activity that changes many elements on a page, you have your hands full just writing the assertions for your expected differences. Everything else should just take care of itself.

Raja’s point in each of these early chapters shows that coded assertions for DOM inspection miss all sorts of behaviors that can vary from browser to browser or app version to app version. Visual assertions with visual AI allow users to simplify their functional test code. He shows why, compared with coded DOM checks, visual assertions provide:

Simpler deployment
Simpler maintenance
More robust test infrastructure

So, in chapter 6, Raja provides this analysis to testing an e-commerce app.

Testing E-Commerce App Elements

Raja begins by pointing out that many of the elements in an e-commerce app behave like elements in his earlier chapters. We find:

A table of catalog elements with regularly placed structural parts, such as product name, description, price, and availability.
Parts of the app screen that depend on previous activity (e.g. recently viewed products)
App behaviors best tested as data-driven (add to cart on an in-stock vs. out of stock item)

With this example, each of the prior chapters in Raja’s course comes into play.

For instance, with the catalog, the legacy approach would require the tester to identify each web element locator in the catalog section, and then ensure that the value in the catalog matched the value in the test code. With the modern approach – take a snapshot. This aligns with chapter 2 of Raja’s course.

With the items in the catalog, a shopper can inspect the details of an element and then click to add it to his or her cart. The shopper can inspect both available and unavailable items – but unavailable items generate an error when the button is clicked to add the item to the cart. Testing this behavior reminds me of chapter 3 in Raja’s course, the chapter about data-driven testing.

Legacy Testing – Assert Element Locators

Raja then walks through the examples and shows the legacy test code. Here is the legacy test code for the catalog test. Note that each output value must be validated.

Basically, for each item in the catalog, check its:

Label
Price
Availability
Color
Image

Compare for each against its expected reply. The code calls out to each web element locator and asserts its value.

Read this code, and then think about your own tests. Consider the similarities and differences. How much of this code remains valid between app upgrades? How much must you rewrite when development adds a new function – or a new feature? Or, if the locators change?

Applitools Simplifies Functional Testing

If you’re with me so far, you understand Raja’s next point – why not leave the assertions to Visual AI? In this case, the assertions let you focus on code that exercises the application and let Visual AI perform the output comparisons.

For testing the tables of inventory, Visual AI lets you perform tasks like sorting and filtering and check the output. For functions like testing what happens when you encounter success versus error conditions – like trying to add an in-stock versus out-of-stock item to the shopping cart, this is the data-driven testing scenario. As he showed in the prior chapters, and again here, Visual AI simplifies the entire test development process.

When you add new behaviors, they become new checkpoints that you can choose to include in the future baseline. And, if you encounter unexpected behaviors, you can reject those changes and send them back to your development team for repair.

As we have discussed elsewhere, visual AI provides the accuracy of AI as used in self-driving cars and other computer vision technology. Visual AI breaks a sea of pixels into visual elements without relying on the DOM to identify those elements. Once you have established a baseline, every subsequent snapshot of that screen will compare the visual elements on that page versus your checkpoint.

Instead of relying on a simplistic pixel comparison to determine whether the checkpoint and baseline differ, visual AI checks the boundaries of the element and determines if the element itself differs in color, font, or – in the case of a photograph or other image file – image completeness.

Rather than depend on DOM differences alone, visual AI compares the rendered page with the previously rendered page.

Visual AI and Packaged Application Upgrades

As I wrote earlier, how does testing a packaged application compare with testing a custom app?

As a test engineer, using the legacy approach leaves you at the mercy of the vendor. You can write all the functional tests you want, but the developer can change element locators between releases and leave you with a huge coding task – rewrite all your element locators to initiate behaviors and all your locators to measure responses.

You likely know the situation I’m describing. I know many companies that use a vendor’s web service for CRM, marketing, or commerce. Once you customize one of these sites for your needs, you worry that a vendor upgrade can break your customized app. The vendor may have tested the generic app, but not the version with your custom CSS, data, and layout.

In my experience, test engineers loathe having to test packaged applications. Too often, test engineers must react to changes made in an app that the company doesn’t own with tools that make it difficult to expose unexpected changes. Imagine having to use legacy tests to validate that the browser rendering remains unchanged after an upgrade. And, if the browser rendering has changed, to highlight that change and pass the information back to the app owners.

Visual AI bridges the gap for a packaged app and third-party app owners. If you have made no change and upgrade your app version, the rendering check with Visual AI makes it easy to test whether the pages have changed. If you make a CSS change or other local or global change, Visual AI ensures that the changes match your expectations.

What About Dynamic Content?

One of the conditions Raja discusses is the case of dynamic content on a page. You find this content on media sites – with layouts showing the latest stories that update regularly. You find this content on e-commerce sites as well – showing wares of interest to prospective buyers.

On the app for this chapter, it’s the content that shows featured items. In the two screenshots that follow, the “Deal of the Day” rotates to multiple images, and “Promoted Items” updates each time the page gets refreshed.

This behavior makes sense – a seller wants to show a range of wares to the shopper.

Both the legacy approach and the modern approach can’t handle dynamic content through automation.

Raja describes how Visual AI allows you to highlight this region as an ignore region when comparing the pages for test automation. My takeaway here says that dynamic content requires a different kind of testing – earlier in the test development process. For the functional behavior test, Applitools Visual AI lets you focus on what matters.

Conclusion – Visual AI Simplifies Functional Test

In walking through Chapter 6, you see how the legacy approach leans heavily on web locators that can change from version to version. You spend lots of resources on maintaining your tests – especially when locators change between app versions. The cost of test maintenance inclines most organizations to shy away from app version upgrades and even app behavior changes. Every time you make a change, you incur unexpected costs.

In contrast, Visual AI simplifies your functional test, while adding visual coverage. Visual AI compares rendered output against rendered output. If visual differences exist, you can identify those as intended or not. Intended changes become the new baseline. Unintended changes can go back to the app owners. And, all this is easy to manage and maintain – because you, as the tester, don’t worry about the individual identifiers you need to check along the way.

Test iFrames With Visual AI

Michael Battat — Mon, 24 Feb 2020 16:37:12 +0000

As you know, I’m taking Raja Rao’s Test Automation University Course, Modern Functional Test Automation through Visual AI. Today we’ll discuss testing iFrames.

In Chapter 5, Raja refers to iFrames as a necessary evil. Actually, they’re really useful. You likely know that you can use iFrames to embed lots of functionality into your page. If you don’t know anything about iFrames, learn a little. There’s this page from the Mozilla Developer’s Network. Or, you can look into a company like Elfsight that provides iFrame widgets you can install to add interactivity to a web page. In some cases, you might use iFrames for tracking and analytics.

Actually, tracking and analytics make up many reasons why you might install iFrames. While they can look seamless on your app, they can call external functions – including third-party tracking tools. But, you can embed almost any functionality using iFrames.

Like, say, a Google Map.

A Quick Google Map Example

The code to embed a Google Map is an iFrame:

Yup, this code will embed a Google Map of Alice Springs, Northern Territory, Australia. Someplace I’ve always wanted to visit. I haven’t made it, yet.

You can use the tracking pieces to know where your customers go on your web app. In fact, some of these tracking iFrames are zero pixel iFrames, meaning they are invisible on your page. The existence of zero pixel iFrames cannot be detected in your end-user and integration testing. They will exist in your DOM, but end users won’t know about their existence.

What matters, then, are the pages that include iFrames for extended functionality. They are embedded in your web app – sometimes nested. How do you test an iFrame?

Possible iFrame Errors

iFrames can be built to the size of the user’s screen, or any subset of your app page. As I mentioned, some are one or zero pixels high. The challenge comes around building interactive iFrames.

When you test iFrames designed for users interaction, the iFrame can behave as if is part of your app. Buttons and functionality inside the iFrame will need to be tested. These iFrames have their own contents, buttons, and scrolling regions. They have their own behaviors that need to be tested.

You will want to test the data and functionality in your iFrames. You will want to ensure that buttons behave as expected, scrolling regions scroll, and expected information gets presented to the user. These behaviors get complicated by how you define your iframe and ensure its boundaries. For some designs, the behavior works fine on a mobile device but gets lost on a desktop or laptop scree. In other designs, the mobile device behavior might not be acceptable.

In this example, Raja points out design considerations that your developers may not have considered when designing your app. The big problems occur with testing iframes sized smaller than their contents – which results in scroll bars on the iFrame to access all the content. Or not – some iFrame implementations are fixed with no scroll bars. How do you test the iFrame?

Testing iFrames with Selenium

Raja imagines a test of a table in a nested iFrame. As the page is resized, the top-level iFrame resizes, and the content in the nested iFrame gets hidden. Depending on the design approach, this hidden data may not become visible unless the user scrolls the data manually. And, in some cases, there may not be scroll bars – meaning a user must manually resize the page to see the data.

Raja walks into the legacy approach to testing iFrames – using Selenium to apply a test behavior and then measure the response of the page. There are several complications he exposes.

Nested iFrame Navigation

The first is the challenge of navigating iFrames – especially if you have nested them. Each time you navigate in an iFrame, you must change the context of the iFrame you’re inspecting. You have to remember your context. And, you have to extract your context to go back to the page outside the iFrame.

In the code that follows, Raja shows the redundant code you need to add to navigate the different nested iFrames. First, you go to the main page context. Next, you go to the top iFrame. Finally, you select one of the four embedded iFrames. Now, you can begin your tests.

Here is a highlight of the code used to change context:

This is a lot of redundant code needed to navigate around to each iFrame.

Hidden Content

Another complication involves the hidden parts of frames. When pages resize and frame content gets hidden, the user may not know the content exists, or its values.. Hidden content can create problems in testing, because the test may not uncover the error.

And, thus, the issue of scroll bars. If a scroll bar does not exist, can the user see data that does not show up on the page? How does a tester use an automated tool to expose this behavior?

Selenium Testing iFrame Application Versions

Selenium can accomplish iFrame resizing and scrolling, but the efforts to do these tasks depend very much on the test engineer. Also, there are dependencies on the version of the application itself. But, validation and assertion technologies limit traditional functional test approaches.

For example, a tester may get a version of the app with scroll bars in a nested iFrame. The engineer might choose not to exercise the scroll bars, and leave that task to the user later. Instead of testing the existence of scroll bars, the tester might simply use text locators to validate that the content exists in the DOM.

If a later version of the app leaves out the scroll bars, then the locators still find the appropriate content, but the user has a much more difficult time accessing the content. Depending on how you structure your Selenium tests, you might miss this change and pass the test. In fact, all functional testing that uses DOM checking will generally miss the user problem this type of issue poses.

Also, things like nested iFrames cause context problems for both applying an action and asserting an outcome. Each time a sub iFrame requires an action, Selenium must first switch context to the main iFrame, then the sub iFrame in question.

Raja describes these types of issues with iFrames as generally problematic for legacy functional tests.

Testing iFrames with Visual AI

Because Visual AI uses image comparison to determine the rendered output, the existence of scroll bars does not require any kind of action or assertion. Instead, if the scroll bars are otherwise invisible, selecting the iFrame should highlight the scroll bars.

In fact, much of iFrame testing with visual AI simply requires taking an action, and then a snapshot of the page. Scroll bars, iFrame contents, and iFrame formatting all can be checked by taking an action along with a snapshot.

Here is a picture of the code from the course.

Complicated assertions become simple implementations of:

eyes.checkWindow(“iFrame”);

Subsequent tests of iFrames can compare visual behavior of the page with its predecessors. Visual AI captures missing scroll bars and other changes in iFrame contents, format, and behavior. You can approve intended behavior changes with a “Thumbs-up”, andy can call out unintended behavior changes (like missing scroll bars and hidden data) with a “Thumbs-down” and a snapshot of both the behavior and the related code.

Because visual assertions replace calls to locators, test engineers have an easier time writing and maintaining test code.

Conclusion

This chapter makes the point that, once again, tasks that can make the life of a test engineer exceedingly difficult with legacy functional tests turn into simple problems for Visual AI. In this case, you don’t have to instrument every iFrame behavior. Simply use Visual AI and let the visual differences guide your testing.

In all honesty, your mileage may vary with this chapter. I know that some applications make heavy use of iFrames for user interaction, while others only depend on iFrames for user tracking. So, depending on the technology you deploy in your applications, you may or may not benefit from this particular value of Visual AI. But, the point Raja makes is that this is yet another task made easier with Visual AI.

How Do You Test Dynamic Content?

Michael Battat — Wed, 12 Feb 2020 17:52:41 +0000

Imagine this. You built a page with CanvasJS, and you want to test the graphs. How do you create an automated test for the graphical representations? It’s testing dynamic content, after all.

This question haunts most test developers. In lots of cases, companies do lots of manual tests on the first release to make sure everything works. After that, it’s a lot of manual spot testing without automation. Because, it’s testing dynamic content, after all.

In reality, there are three approaches.

Always do manual testing – that’s the only way to validate behavior.
Do spot testing – trading off coverage for the cost.
Shy away from testing and hope things work.

At the radical end of handling dynamic graphical content, some organizations decide that the job doesn’t belong to the internal web development team. These organizations farm out the entire visualization process to a third-party graphing package. For example, the Federal Trade Commission concluded that their data is best visualized using an external solution focused on data graphics and analytics. As a result, they use the services of Tableau Software to create a visual representation of the FTC data.

But, if you’re building an app for which farming out data representation might expose customer or client data, you cannot give the data to a third party and hope for the best. You have to do the visualization and own the testing of the app.

Dynamic Content Tests With Legacy Tools

In Chapter 4 of Raja Rao’s course, Modern Functional Test Automation through Visual AI on Test Automation University, Raja walks through an example graphing app built with Canvas and asks:

“How would you test this page with dynamic content?”

He takes a bar chart example in an app using CanvasJS.

Next, Raja shows what happens when he adds a dataset to the bar chart:

What makes this problem notoriously difficult to test involves the visual nature of the behavior and the lack of handles in the DOM that correlate to the behavior. In fact, there are no links.

Opening up the Inspector for this page shows a canvas link:

So, all it shows is the size of the Canvas render – not the internal content. How the heck do you test this dynamic content?

With no DOM hooks, it’s impossible to know that the code above behaves as expected.

How would you handle this kind of test? When we ask, we find out that most people do is either test on occasion or not at all. After all, if you’re using a third-party package, like CanvasJS, why not just trust it and go?

Testing Charts with Visual AI

As Raja points out, with Visual AI, you don’t need hooks in the DOM to capture app behavior. All you need to do is trigger the behavior, then capture the results visually.

Here is the test code he uses to manipulate the test chart:

Hopefully, each step in the code reads clearly for you:

Open the app
Capture the screen
Click the add dataset button
Wait to make sure the screen executes
Capture the screen

It seemed pretty straightforward to me when I went through it.

When you run the code, Applitools captures the tests separately as part of the same batch test run:

When you start running Applitools on these tests, the first runs get stored as the baseline expected images. You can continue to execute these tests on subsequent builds and have Applitools compare the new checkpoint against the baseline. Applitools will highlight any visual differences.

Why Does Testing Dynamic Content Matter?

In the past, I have been responsible for apps that display lots of data – like the central controller for a bunch of networking equipment. Lots of data and visualization. Each time we thought about improving the visualization, it was a huge headache. Testing alone would swallow up the QA team in apoplectic fits.

In looking at the world of visualization, there are network operations centers.

There are financial applications.

There’s even weather.

Whether you’re handling inventory, forecasting the future, scheduling appointments or doing any number of things with your applications, your customers likely will benefit from data visualizations. Why let the question of test automation limit your decision of whether or not to deploy a great visualization?

Conclusion

Everyone who deals with data needs to represent that data as more than a bunch of numbers. If you find yourself doing visual representations, you have a choice:

Code, test, and pray
Code, test, and spot check
Test visually and automate tests of dynamic content.

Now that you have a way to test this dynamic content, what’s stopping you?