JavaScript failed your tests

In this article, we'll discuss how a harmless-looking line of JavaScript code caused test instability in a product and how to prevent such issues.

Our static analyzer integrates with many tools, including IDEs, allowing developers to easily use it throughout the development workflow. One such integration is the Visual Studio Code extension in JavaScript and TypeScript.

Note. To learn more about how to use PVS-Studio plugin for Visual Studio Code, please see the documentation.

As with any other product component, we maintain the extension quality by leveraging various approaches to verify and test it. UI tests that check the main scenarios of user interaction with the plugin are one of them.

At one point, these tests faltered for no apparent reason—that's how we saw it because we didn't touch the code at that moment.

On the case with unicorns

Let's start our little investigation by examining the failure.

The Visual Studio Code Command Palette provides access to many commands within the plugin and can be opened with Ctrl+Shift+P. Thus, our testing framework can interact with this Command Palette.

I mention this because the screenshots saved in CI/CD after the failed tests showed that the Visual Studio Code Command Palette had opened, but nothing else happened:

We'd pinpointed the problem area, but the exact issue was still unclear.

After a long search, we discovered that the testing framework was the failure cause.

Here's the method that interacts with the Command Palette:

async openCommandPrompt(): Promise<QuickOpenBox | InputBox> {
  const webview = await new EditorView()
                    .findElements(
                      EditorView.locators.EditorView.webView 
                    );
  if (webview.length > 0) {
    const tab = await new EditorView().getActiveTab();
    if (tab) {
      await tab.sendKeys(Key.F1);
      return await InputBox.create();
    }
  }
  const driver = this.getDriver();
  await driver.actions()
          .keyDown(Workbench.ctlKey)
          .keyDown(Key.SHIFT)
          .sendKeys('p')
          .perform();

  if (Workbench.versionInfo.version >= '1.44.0') {
    return await InputBox.create();
  }
  return await QuickOpenBox.create();
}

In this method, the framework obtains the necessary objects and emulates the required key combination using the web engine. Then, depending on the Visual Studio Code version, it creates objects to which the necessary commands will be passed: InputBox is for later versions and QuickOpenBox is for earlier ones.

Everything looks pretty good, but a single code line caused the failure:

...
if (Workbench.versionInfo.version >= '1.44.0') {...}
...

Here, we're comparing versions to determine which object to use for interacting with the Command Palette. Note that the version is specified in a string.

Now, let's check the latest version on the Visual Studio Code website:

Let's try making a comparison with this value:

console.log("1.105.0" >= "1.44.0") // false

Since the version is stored as the string, we compare the numbers lexicographically. So, we're moving from left to right, character by character, and comparing the Unicode character codes.

So, in the code snippet above, TypeScript compares 1 and 4 and draws the conclusions because it doesn't know that 1 has higher precedence than 4.

As a result, the framework selects the incorrect approach for passing commands to the Visual Studio Code Command Palette, which caused the tests to freeze.

We could fix this by storing the version in an object instead of a string:

interface VSCodeVersion {
  major: Number,
  minor: Number
}

This issue has been fixed in later framework versions by using a separate library for version comparing.

Dynamic typing

You might say, "The title claims JavaScript failed the tests, but this could've happened in any language—even one with static typing." And you would be absolutely right. However, in JavaScript, there's a very specific reason behind such incidents.

It's a dynamically typed language, so strange things can sometimes happen with its types.

For example, if we want to accurately compare two values, we need to use the === operator. It also checks object types for equality.

On the other hand, the more familiar == operator performs type conversion before comparison, meaning that even values of different types can be considered equal.

For example, we can compare true and 1:

console.log(true == 1);   // true 
console.log(true === 1);  // false

Since type conversion makes true equal to 1, the comparison result will be true.

We can compare numbers and strings in the same way:

console.log("5" == 5);   // true
console.log("5" === 5);  // false

TypeScript is a JavaScript superset that adds static typing. Depending on the settings, however, the TypeScript compiler may not issue warnings for such comparisons.

A static code analyzer, for example, can help identify these kinds of issues in a project.

Conclusion

This story is a reminder of how easy it is to overlook "obvious" details like string comparison—especially in a language where types don't always behave the way you expect. Even in projects with seemingly strict typing, such pitfalls remain possible.

By the way, there's a good reason why we published this article about JavaScript on the PVS-Studio blog. Who knows, maybe PVS-Studio will introduce an analyzer for JavaScript and TypeScript soon...

Clean code to you, folks!