A Modern Approach to Manual Testing

Written by Juan Fernandez

Setting the stage

We’ve all seen it, the bigger an organization gets, the more difficult it becomes to have meaningful and efficient communication between teams. This is particularly apparent when incentives are misaligned. Often, development teams are incentivized to build as much as possible in the shortest time possible. On the other hand, QA teams are incentivized to reduce the inherent risk in a codebase/application to an acceptable degree. The problem is obvious: whereas development teams optimize for speed, QA teams optimize for correctness. This misalignment of incentives results in frustration and burnout between and within these teams.

This inherent friction between development and QA teams is most acute when finding a bug. At that time, QA professionals prioritize having said bug fixed. In contrast, developers get conflicted between shipping a new feature in a timely manner, or delaying it in exchange for a bugfix, which could take five minutes or five weeks to implement, and has no clear ROI.

While an argument could be made that the right culture could [potentially] prevent this, our experience having talked to hundreds of organizations is that, more often than not, teams with competing incentives behave this way.

The state of affairs

Let’s take a look at what happens today, at any given organization, when QA finds a bug in an application. For the sake of this post, let’s pretend this is an e-commerce application.

Step 1: QA contacts the developer or development team supposedly responsible for the fix. They do this on a communications tool like Slack, or by creating an issue on a tracker like Jira.

Step 2: The QA professional then proceeds to talk with the “Checkout Dev Team.”

Step 3: QA is left confounded and unsure of how to proceed.

What would follow is a back and forth between the QA and development teams struggling to reproduce the issue across different development environments, while sharing screenshots and recordings, and trying to fish for the appropriate logs of the apparent issue at hand. All of this while banging their heads against the wall, because the problems only show up under seemingly random circumstances that cannot be determined. On top of it all, it’s likely that after days with this issue lingering and bouncing around JIRA, not one development team confidently claimed ownership of the fault, let alone is working on a fix.

This sequence of events happens even in the most dedicated and collaborative organizations with robust processes in place. The problem is the depth and complexity of the software applications that we’re developing these days. Correctly identifying what team, microservice, configuration, or transaction is at fault for any given issue, and quickly debugging it to find a proper solution is one of the most difficult challenges in modern-day software engineering.

Can we do better?

Of course, we can! But before I show you how, let me introduce myself: I’m Juan, and I work as a Software Engineer at Undefined Labs. We work on developer tools, and we believe it’s time to fundamentally change the way testing and development teams collaborate, and significantly improve how organizations test their applications. In this post, I’d like to show you the power of Scope for manual web-based testing.

To explain our approach better, let’s revisit our previous scenario, this time with Scope:

QA can provide a deep link to the test report for Developers to analyze.

All reports in Scope include a trace showing the transaction recorded during the execution of the test.

The verdict: screenshots and logs, while handy, are hardly sufficient to deal with most of the issues a development team will face any given day in today’s modern applications. With Scope, we’re making it trivially easy for QA teams to provide their development teams with the rich and in-depth visibility they need to fix any manual test and any bug. Let’s see how.

Introducing Scope for Chrome

When we set out to build Scope, we mainly set out to solve problems for developers. And while developers are ultimately responsible for fixing a bug, we saw in Scope the potential to build a bridge between QA and development teams. Scope for Chrome is our first step towards building this bridge.

Whereas many of the frameworks, SDKs, and agents for testing we’ve built to date cater to developers, it was clear early on that the needs and tools of a QA are quite different. With Scope for Chrome, we wanted to build the easiest way for anyone to create meaningful manual browser tests. As such, the only requirement to use Scope for Chrome is to (1) install our browser extension, and (2) know how to use a web browser.

It really is that easy:

After anyone records a manual test using Scope for Chrome, a report is generated automatically. This test report includes:

• User actions like clicks or keyboard strokes.

• HTTP requests with their headers and payloads.

• Responses by the backend and even database queries being fired as a result of the request.

• Console logs.

• Exceptions.

Finally, every manual test comes with a detailed report, including everything a developer would want to see when trying to debug any given regression. And best of all, it eliminates the need to try to reproduce the issue at hand. There is also no “what browser were you using?” or “what user were you logged in as?” Everything you need to understand the problem is in a single pane of glass. And it is easier than ever to know what team is best suited for the fix, or what service is at fault.

How it works

Simply put: when a user clicks on the “Start Recording” button, Scope For Chrome starts listening and recording everything happening within your current tab.

In more detail, to accomplish this, our effort is threefold:

• Listen to and record user events such as mouse clicks, keyboard strokes, and exceptions happening within the tab.

• Monkey patch XHR and fetch by injecting code to the tab under test. Each request creates a new span (“individual unit of work” as per OpenTracing terminology) that then propagates to the backend.

• Listen to main_frame requests (a document that is loaded for a top-level frame). This is the first request that your browser does when going to a new page. For this, we use the webRequest extension API.

Of course, this is just a glimpse at what Scope for Chrome does. None of it would have been possible without the work behind our javascript agent, our other agents (Python, iOS, .NET, Java, Golang), our backend capable of ingesting and processing test data and distributed traces, and our purpose-built web UI to display tests in an interactive and structured way.

P.S. Check out the Technical Addendum below for more information.

What’s next?

With the ever-increasing complexity of modern software comes bigger, more sophisticated software teams. In the same way teams work to improve the interfaces between distributed services, we should improve how we collaborate and communicate between teams. We believe Scope for Chrome can help alleviate the most frustrating problems associated with the lack of visibility in manual testing, and help bridge the gap between QA and Dev.

You can learn more about Scope for Chrome here.

Technical Addendum: technical challenges faced while building Scope for Chrome.

1. Monkey patching is hard

For those that haven’t dwell into the world of monkey patching, it is a technique to add, modify, or suppress the default behavior of a piece of code at runtime without changing its original source code.

For example, consider a class with a method get_value. Suppose this method does a database query when being called. Imagine you are now unit testing your class: you may not want to do an actual database query, so you dynamically replace get_value by a stub that returns some mock data.

This can be extended to other uses. For example, in a web application, you might want to substitute console.log by a function that not only logs a message but also adds the date at which the function was called.

Here’s an example:

const log = console.log

console.log = function() {

  log.apply(console, [new Date().toISOString(), ...arguments])

}

Basic monkey patching of window.fetch is simple (note that this is just an example and is not prepared to be production code):

const oldFetch = window.fetch

window.fetch = (...args) => {

   // do something here
   return oldFetch(...args)

}

Things get more interesting when you want to do async stuff in there, like communicating with a background script:

const oldFetch = window.fetch

 window.fetch = (...args) =>

  new Promise(resolve => {

   asyncCommWithBackground().then(newRequestInfo => {

    const newFetchArgs = [...args, ...newRequestInfo]

    resolve(oldFetch(...newFetchArgs))

   })

  })

This pattern is quite powerful. But with great power comes great responsibility. By monkey patching, we are slowing every request in the active tab down by however long asyncCommunicationWithBackground takes to resolve.

And here’s an example of doing something with the result of the fetch:

window.fetch = (...args) =>

 new Promise(resolve => {

  asyncCommWithBackground().then(newRequestInfo => {

   const newFetchArgs = [...args, ...newRequestInfo]

   resolve(oldFetch(...newFetchArgs)).then(fetchResult => {

    // do something with fetchResult

    return fetchResult 

   })

  })

 })

}

You can simplify the code a bit with async/await, but you have to be careful. You probably want to know if your fetch has failed, for which you would use try/catch. But if you do that, you would stop exceptions from propagating to the consumer, which is a scenario you’d want to avoid. The most important thing to remember here is: monkey patching done right should be transparent.

To do something with the response data, like adding it to the span as tag or metadata, you need to be careful with cloning your response before, as it is a stream and can only be consumed once.

If you want to dig a bit deeper into this pattern, we have set up a repository with a small project of a functioning chrome extension that delays all your fetch requests. The pattern is highlighted here.

An alternative to this solution is to use the webRequest API with hooks like onBeforeSendHeaders, but as of now, this API does not allow the capture of response payloads, which was a requirement for Scope.

2. Monkey patching in a browser is even harder

As it turns out, if you want to affect the window variable of the tab, which we need for monkey patching, a content script is not enough. You need to execute code like the one shown in here. This means handling your code as string. There are some alternatives like using function.toString() and babel macros to evaluate variables in build time, but the extra complexity defeats the purpose, as your monkey patched functions should not be big anyway.

Utility functions that your monkey patched functions need, like random number generation or parsing of data need to be available in the tab, which means again handling your code as strings to inject it. The tab shares no execution environment with your background, and while it would be possible to asynchronously request and wait for results, this would mean slowing down all your requests.

3. Sending responses asynchronously

Your content scripts and injected code will communicate with your background through message passing. At some point, the responses might require some async operation. To leave the communication channel open you need to return true before calling sendResponse. More of this pattern here.

4. Bundling your extension

The window that appears when you click on your browser’s extension icon, also known as a popup, will grow sooner than you expect and a state management library will come in really handy.

To avoid the hassle of managing a webpack configuration with all the perks (like hot reloading), there are some excellent starter projects and tools for this specific purpose.

samuelsimoes / chrome-extension-webpack-boilerplate

A basic foundation boilerplate for rich Chrome Extensions using Webpack to help you write modular and modern Javascript code, load CSS easily and automatic reload the browser on code changes.

Chrome Extension Webpack Boilerplate

A basic foundation boilerplate for rich Chrome Extensions using Webpack to help you write modular and modern Javascript code, load CSS easily and automatic reload the browser on code changes.

Developing a new extension

I'll assume that you already read the Webpack docs and the Chrome Extension docs.

1. Check if your Node.js version is >= 6.
2. Clone the repository.
3. Install yarn.
4. Run yarn.
5. Change the package's name and description on package.json.
6. Change the name of your extension on src/manifest.json.
7. Run yarn run start
8. Load your extension on Chrome following
   1. Access chrome://extensions/
   2. Check Developer mode
   3. Click on Load unpacked extension
   4. Select the build folder.
9. Have fun.

Structure

All your extension's development code must be placed in src folder, including the extension manifest.

The boilerplate is already prepared to have a popup, a options page and a background page. You can easily customize…

View on GitHub

xpl / crx-hotreload

Chrome Extension Hot Reloader

Chrome Extension Hot Reloader

Watches for file changes in your extension's directory. When a change is detected, it reloads the extension and refreshes the active tab (to re-trigger the updated scripts).

Here's a blog post explaining it (thanks to KingOfNothing for the translation).

Features

• Works by checking timestamps of files
• Supports nested directories
• Automatically disables itself in production
• And it's just a 50 lines of code!

How To Use

1. Drop hot-reload.js to your extension's directory.

2. Put the following into your manifest.json file:

    "background": { "scripts": ["hot-reload.js"] }

Also, you can simply clone this repository and use it as a boilerplate for your extension.

Installing From NPM

It is also available as NPM module:

npm install crx-hotreload

Then use a require (or import) to execute the script.

View on GitHub

We’ve used React but these should work for any other state management library.

5. Browser compatibility

The problem with browser compatibility is not well solved with extensions. Though we have not dug into it yet, there seems to be a lot of potential in web-ext.

6. Host commands

Our javascript agent gets some of its metadata with host commands. But you’re running in a browser extension, so that is not an option.

Some questions then arise:

1. Where to get credentials from? Traces sent to the backend need an API endpoint and an API key.

2. How would I calculate the NTP offset in my machine needed for precise timestamp measurements in the trace view? When talking about distributed traces, resolution and precision in the order of microseconds and even nanoseconds is important, as the traces are often generated in different machines. Any small offset can ruin your data.

3. In the future: how do I get the code of this specific file and line number that threw an exception?

There are different ways to answer these questions. For example, we can solve number 1 by logging our extension in using the browser’s cookies. This would allow the extension to send authenticated requests to our backend. This is risky though, as it requires a SameSite=None cookie. Number 2 is tricky as there are no “external world” solutions — we need host rights. Number 3 could be solved the same way as number 1, but again, that is risky.

An option that answers all 3 questions is using our Scope Native App via the native messaging API. The native app is already configured with the API endpoint of your choice and it has access to the API key, which solves question number 1. It can also run host commands, so that solves number 2 and number 3. The disadvantage is that we couple our extension with a different product, but with our current and future requirements in mind, this seems like the best possible alternative.

Building browser extensions is very rewarding. The technical challenges we faced on this one were really thought provoking and and we’re sure we will continue to face many more. You can also rest assured we’ll continue to invest and innovate in this space, as we’re confident of our ability to help bridge the gap between Dev & QA with tools like Scope for Chrome.

Testing is a core competency to build great software. But testing has failed to keep up with the fundamental shift in how we build applications. Scope gives engineering teams production-level visibility on every test for every app — spanning mobile, monoliths, and microservices.

Your journey to better engineering through better testing starts with Scope.