DEV Community: Symflower

Isolating IntelliJ plugin tests using temporary project directories

Symflower — Mon, 16 Jan 2023 08:40:17 +0000

When testing a plugin for IntelliJ-based IDEs using a UI test, the plugin gets to operate within a fully-featured instance of the IDE. Most tests will therefore require a project to perform their actions on. This opens a question: How do you deal with modifications to test projects made by tests? Add undo logic at the end of every test? Revert using a version control system? Those options sound like easy ways to make new mistakes.

We faced the same problem when writing tests for our own plugin, Symflower for IntelliJ IDEA and GoLand, but we've opted for a simpler solution. We don't run tests on the canonical source of our test projects but rather copy the entire project directory and use that copy instead. This avoids relying on repetitive cleanup logic or VCS-based reverting that could undo legitimate changes to the source files.

Curious about how we do it? Head over to Symflower's blog to read the rest of this article:
Isolating IntelliJ plugin tests using temporary project directories

Accessing IntelliJ plugin classes in UI tests

Symflower — Tue, 10 Jan 2023 11:29:47 +0000

When testing an IntelliJ plugin, you'll likely reach for UI tests to verify that both the user interface as well as the code behind it functions correctly. In UI tests, your plugin runs in a real instance of IntelliJ that is similar to a production environment on a user's machine. That increases your end-to-end coverage but sometimes it's harder to verify success conditions for a test in that environment. So as a last resort, you might want to call into a service that's part of the plugin to verify some internal state directly.

For example, we found for our Symflower plugin, we needed this to check for progress indicators displayed by our plugin. We could not verify our plugin's progress indicators purely from the UI because they were too fast for the UI automation to pick them up. So we had to build a service into our plugin that, when run under test, keeps references to shown progress indicators to allow tests to verify them. But how would one go about doing that?

To find out, read the rest of this post on Symflower’s blog!

What is new in Java 20?

Symflower — Thu, 24 Nov 2022 15:10:03 +0000

Java 20 is due for release in March 2023, and is expected to introduce a range of changes and new functionality. We prepared a sneak peek into which JEPs are most likely to be accepted into JDK 20, and which ones we are excited to hopefully be accepted next!

The latest version of the JDK (Java Development Kit) 19 was released on the 20th September 2022. The next version, Java 20, is planned as a non-LTS release, while the following version 21 is set to be a release with Long-Term Support (LTS). The upcoming version 20 is expected to bring a number of great updates, so we're excited to get our hands on it in March!

But before we jump into the JEPs lined up for Java 20, let's do a quick overview of the process for updating Java and the concept of enhancement proposals so that you have the full picture on what's going on.

JDK Enhancement Proposal (JEP) & roadmap process

Development of the Java Development Kit (JDK) is based on the concept of JDK Enhancement Proposals, or JEPs. In essence, these enhancement proposals serve as the roadmap for JDK release projects and all related development activities. The JEP process does not replace the Java community process as that is still the primary way to approve changes in the Java API and language. Rather, Oracle's intention for introducing the JEPs back in 2010 was to enable OpenJDK contributors to submit ideas to improve the ecosystem. JEPs help track the development status of features intended for delivery in a JDK release project, as well as work that is not linked to a specific release.

At the time of writing, the JEP Index lists 437 enhancement proposals, as well as a number of draft and submitted JEPs. Of all these enhancement proposals, the first 4 are meta-JEPs:

JEP 0 is the JEP Index of all posted proposals referenced above.
JEP 1 outlines the process for managing (collecting, reviewing, prioritizing & evaluating, and recording the results) of proposals for JDK enhancement, process, and infrastructure improvements.
JEP 2 is a template to be used for creating new JEPs.
JEP 3 describes the JDK release process.

The output of the JEP process is a JDK roadmap that contains proposals for new features as well as other improvement ideas to be considered for inclusion in JDK release projects. The roadmap basically serves as a backlog, covering ideas for at least the next three years, so not all JEPs will make it into the next release.

Source: http://cr.openjdk.java.net/~mr/jep/jep-2.0-fi.png

You can check out the current list of JEPs (drafts and submitted candidates) for Java 20 on the JEP 0, as well as the Early-Access Release Notes for changes of this version. OK, now that you understand the process, let's dive right into what JEPs are expected to make it into Java 20!

Read the rest of this post over at Symflower's blog!

Software Testing Trends for 2023

Symflower — Thu, 17 Nov 2022 14:39:09 +0000

With software becoming the main value driver and the primary interface between developers and customers, focus on software quality is growing. The need to deliver high-quality, bug-free products in ever-accelerating cycles is putting pressure on software teams. Software testing has to apply new methods to keep pace. Here's a list of new testing strategies that are gaining ground in 2023, and which software testing trends Symflower expects to continue to unfold this coming year!

Continue reading on Symflower's blog:
Software testing trends for 2023

Software Development Trends for 2023

Symflower — Wed, 16 Nov 2022 20:44:25 +0000

As software is taking over the world, the development strategies we use have to evolve in lockstep to keep pace. With development teams constantly looking to innovate and improve their efficiency, there are a few defining software development trends emerging that will influence how we deliver software in 2023 and beyond.

To find out about the key trends defining software development in 2023, continue reading on Symflower's blog:
Software development trends for 2023

Java Fuzzing with Jazzer compared to Symflower

Symflower — Tue, 15 Nov 2022 11:23:36 +0000

Fuzzing is a testing technique where random values are generated as inputs to find unexpected behavior such as crashes and security issues. Previously we looked at the new Golang release 1.18 which includes native fuzzing and our Core Technology blog series contained a post that compares symbolic execution to different other testing techniques, including fuzzing. Today, we will dive into the Java world and check out the most popular Java fuzzing solution: Jazzer.

We will explore how Jazzer is used to automatically generate malicious inputs for Java programs, and how it compares to Symflower, which can automatically generate unit tests to uncover bugs and errors in your code. With the help of Jazzer, many bugs - some of them even in the OpenJDK - were found already. Also, as of March 2021, Jazzer is officially part of OSS-Fuzz, Google's cloud fuzzing engine. It should be noted that Jazzer is a pure "bug detection" utility that finds reproducers for errors in user code. Symflower can do the same, but provides additional functionalities to boost developer productivity, like generating high coverage unit tests and providing test templates for the software developer or tester.

Tool Setup

We will now go through the process of installing both Symflower and Jazzer so you can easily follow along on your local machine. First, we create a new folder fuzz where we can place everything we need.

For Symflower, we can just head over to get.symflower.com to install either the CLI version or an IDE plugin. No matter which version we choose, we just need to run the installer and are immediately good to go.

To install Jazzer, we need to download the executable and its dependencies from the GitHub releases page. We then extract jazzer, jazzer_agent_deploy.jar and jazzer_api_deploy.jar into our fuzz folder to have them at the ready. For a permanent installation, we'd need to move these files to where our PATH and Java can pick up on them.

Example 1: Validating a username

Let's look at a very simple example first, to get a feel for how the different tools are used. We assume that we have some user database or login system and want to make sure that the usernames consist of only lowercase letters. From this snippet we can see that we just loop over all characters in the input string and throw an IllegalArgumentException when we encounter an invalid character.

// fuzz/example1/Strings.java

public class Strings {
    public static void validateName(String in) throws IllegalArgumentException {
        for (int i = 0; i < in.length(); i++) {
            char c = in.charAt(i);
            if (c < 'a' || c > 'z') {
                throw new IllegalArgumentException("name must consist of letters");
            }
        }
    }
}

Jazzer has an "autofuzz" feature that automatically scans the code one wants to analyze and properly matches the input data types to the program input. In this case we will only tell it that we want to analyze our validateName function and it will automatically figure out that it has to generate random strings as inputs. One downside of Jazzer's "autofuzz" mode, however, is that it only detects unexpected Exceptions from the user code. Here though, we actually expect that there are problems with invalid names since we added a throws IllegalArgumentException to the method body.

This tells Jazzer that IllegalArgumentExceptions are fine for us, so no test cases fulfilling the condition c < 'a' || c > 'z' will be generated. As the behavior of only looking for unexpected exceptions cannot be deactivated for Jazzer, we must add a little hack to actually provoke full coverage of the function.

((Object)null).hashCode();
// throw new IllegalArgumentException("name must consist of letters");

To analyze this version with Jazzer then, we first need to compile the code with javac fuzz/example1/Strings.java, then we can execute the fuzzer by running ./jazzer --cp=$PWD/fuzz/example1 --autofuzz=Strings::validateName. We will see that two files Crash-<hash> are immediately generated, and the fuzzer keeps going, generating more and more random input strings.

public class Crash_b6334b553dfdbd68c65c21aa59b75ae8dafc02ca {
  public static void main(String[] args) throws Throwable {
    Strings.validateName(".");
  }
}
public class Crash_da39a3ee5e6b4b0d3255bfef95601890afd80709 {
  public static void main(String[] args) throws Throwable {
    Strings.validateName((java.lang.String) null);
  }
}

This shows that Jazzer successfully generated an input that provokes the exception, and additionally found the case where a "null" string is used. After half a minute, the fuzzer gives up to generate more values and terminates with an "out of memory" message. Many more random strings were constructed but none resulted in additional interesting behavior.

With Symflower, we can just run the tool without any additional modifications necessary. This is done by either executing the symflower command in the console in our "fuzz/example1" directory, or - if we have an IDE plugin installed - triggering Symflower right from your IDE. The automatic unit test generation takes roughly a few seconds and results in a file StringsSymflowerTest.java.

@Test // (expected = NullPointerException.class)
public void validateName1() throws IllegalArgumentException {
    String in = null;
    Strings.validateName(in);
}

@Test
public void validateName2() throws IllegalArgumentException {
    String in = "";
    Strings.validateName(in);
}

@Test // (expected = IllegalArgumentException.class)
public void validateName3() throws IllegalArgumentException {
    String in = "\u0001";
    Strings.validateName(in);
}

@Test
public void validateName4() throws IllegalArgumentException {
    String in = "z";
    Strings.validateName(in);
}

@Test // (expected = IllegalArgumentException.class)
public void validateName5() throws IllegalArgumentException {
    String in = "\u00ef";
    Strings.validateName(in);
}

The Symflower result also includes the cases where a non-letter and a "null" string is present. Though, since these are unit tests, we also have cases that don't provoke any error but simply exercise our whole example code. While Jazzer found the errors too, Symflower additionally computed a complete test suite for our example with additional information on the outcomes.

Jazzer found cases for the problems very fast. However, it continued trying to find additional ones, even though there are none. Symflower on the other hand finished fast with this example. However, it could be faster. We are working hard on improving the performance of Symflower, so if you find some example that takes a while to process, please let us know through our public issue tracker.

Example 2: Modeling a simple train station

Now, we present a more complex scenario where we model a train station and different types of trains. We check if the station can accept a request to have a train stop, and depending on the type of the train, some conditions have to apply for the train to be allowed to stop at the station.

public class Train {}

public class FreightTrain extends Train {
    int weight;
}

public class PassengerTrain extends Train {
    int length;
}

public class Station {
    int weightLimit;
    int lengthLimit;

    public void accept(Train t) throws IllegalArgumentException {
        if (t instanceof PassengerTrain) {
            PassengerTrain pt = (PassengerTrain)t;
            if (lengthLimit > pt.length) {
                throw new IllegalArgumentException("train cannot safely stop within station borders");
            }
        } else if (t instanceof FreightTrain) {
            FreightTrain ft = (FreightTrain)t;
            if (weightLimit > ft.weight) {
                throw new IllegalArgumentException("train cannot safely use the station tracks");
            }
        } else {
            throw new IllegalArgumentException("unknown train type");
        }
    }
}

As before, we need to manually provoke a "NullPointerException" for Jazzer to pick up on our unwanted states.

((Object)null).hashCode();
// throw new IllegalArgumentException("<error message>");

We note that for Jazzer's "autofuzz" functionality, it is necessary that all of these classes are public, so they need to be placed into their own respective files in fuzz/example2. Then, we must again compile our code with javac fuzz/example2/*.java before we can start the fuzzer with ./jazzer --cp=$PWD/fuzz/example2 --autofuzz=Station::accept. The fuzzer starts generating values, but won't find the more interesting critical inputs any time soon. However, we see that the simple "null" case is present:

public class Crash_adc83b19e793491b1c6ea0fd8b46cd9f32e592fc {
  public static void main(String[] args) throws Throwable {
    (((java.util.function.Supplier<Station>) (() -> {Station autofuzzVariable0 = new Station(); return autofuzzVariable0;})).get()).accept((Train) null);
  }
}

In the meantime (in another terminal window) we can check how Symflower performs. We just run symflower in the fuzz/example2 directory, or - if we're working with an IDE plugin - trigger the analysis from within the IDE. Within seconds, we obtain the following test cases.

@Test // (expected = IllegalArgumentException.class)
public void accept1() throws IllegalArgumentException {
    Station s = new Station();
    Train t = null;
    s.accept(t);
}
@Test
public void accept2() throws IllegalArgumentException {
    Station s = new Station();
    Train t = new FreightTrain();
    s.accept(t);
}
@Test
public void accept3() throws IllegalArgumentException {
    Station s = new Station();
    Train t = new PassengerTrain();
    s.accept(t);
}
@Test // (expected = IllegalArgumentException.class)
public void accept4() throws IllegalArgumentException {
    Station s = new Station();
    s.lengthLimit = 1;
    Train t = new PassengerTrain();
    s.accept(t);
}
@Test // (expected = IllegalArgumentException.class)
public void accept5() throws IllegalArgumentException {
    Station s = new Station();
    s.weightLimit = 1;
    Train t = new FreightTrain();
    s.accept(t);
}

We can see that Symflower found both interesting cases, a freight train that is too heavy and a passenger train that's too long, but also generated tests for the remaining cases.

Jazzer however, seems to struggle with the custom data types we introduced. We need to aid Jazzer with these data types by writing a custom test driver fuzz/example2/Fuzzer.java. For this, we must pick which train type we want to check, so we choose the PassengerTrain. If we want to check both cases, we would need to add another test driver. But one case should suffice to demonstrate the process.

import com.code_intelligence.jazzer.api.FuzzedDataProvider;
public class Fuzzer {
    public static void fuzzerTestOneInput(FuzzedDataProvider data) {
        Station s = new Station();
        s.lengthLimit = data.consumeInt();
        PassengerTrain pt = new PassengerTrain();
        pt.length = data.consumeInt();
        s.accept(pt);
    }
}

To compile the code now, we need to add the Jazzer API javac -cp ".:./jazzer_api_deploy.jar" fuzz/example2/*.java. We don't use the "autofuzz" feature anymore, but directly point Jazzer to the test driver using ./jazzer --cp=$PWD/fuzz/example2/ --target_class=Fuzzer. This time we instantly obtain a crashing input, but only for the "PassengerTrain" case:

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
public class Crash_adc83b19e793491b1c6ea0fd8b46cd9f32e592fc {
    static final String base64Bytes = String.join("", "rO0ABXNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAACdwQAAAACc3IAEWphdmEubGFuZy5JbnRlZ2VyEuKgpPeBhzgCAAFJAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAApzcQB+AAIAAAAAeA==");
    public static void main(String[] args) throws Throwable {
        ClassLoader.getSystemClassLoader().setDefaultAssertionStatus(true);
        try {
            Method fuzzerInitialize = Fuzzer.class.getMethod("fuzzerInitialize");
            fuzzerInitialize.invoke(null);
        } catch (NoSuchMethodException ignored) {
            try {
                Method fuzzerInitialize = Fuzzer.class.getMethod("fuzzerInitialize", String[].class);
                fuzzerInitialize.invoke(null, (Object) args);
            } catch (NoSuchMethodException ignored1) {
            } catch (IllegalAccessException | InvocationTargetException e) {
                e.printStackTrace();
                System.exit(1);
            }
        } catch (IllegalAccessException | InvocationTargetException e) {
            e.printStackTrace();
            System.exit(1);
        }
        com.code_intelligence.jazzer.api.CannedFuzzedDataProvider input = new com.code_intelligence.jazzer.api.CannedFuzzedDataProvider(base64Bytes);
        Fuzzer.fuzzerTestOneInput(input);
    }
}

The test case is quite complex, since Jazzer models both input values as a random string. This string is then converted to the two input integers for the length limit and train length, but we cannot be sure which actual integer values are chosen here.

So as we see, there are some caveats when working with Jazzer and custom data types, requiring manual work to supply the fuzzer with the correct test drivers, and being able to understand the generated test values. Symflower on the other hand generated a complete test suite fast and presents easy to understand test cases. However, we are still not done: if you find code that does not work for you or has results that could be done better, please let us know through our public issue tracker.

By the way, there is an additional bug in the examples above. Were you able to spot it? If so, how? By looking at the code or the test case?

Wrap Up

Symflower is an easy-to-apply, out-of-the-box solution that generates unit test suites but also increases developer productivity through functionalities like test templates. Jazzer, on the other hand, specializes on efficiently generating input values to trigger bugs and errors in user code. Both of these tools are free and focus on improving your development experience - so why not just use both of them? 🤔

We are working hard on providing the best overall solution for generating inputs for bug detection and unit testing. And, we're improving Symflower every day so feel free to check it out and report your experience back to us - we're always happy to receive your feedback!

To be notified of any new blog posts regarding coding, testing, features of Symflower or the memes, feel free to subscribe to our newsletter. You can also follow us on social media and - if this post helped you - share it via Twitter, LinkedIn or Facebook.

Integrating a Language Server (LSP) in a VS Code Extension

Symflower — Thu, 10 Nov 2022 13:28:21 +0000

If you’re working on developer assistance tooling such as auto-completion or error highlighting for a particular programming language, you probably want the functionality with more than one code editor. In the bad old days this used to be quite labor-intensive: developers had to come up with ways to share common logic across different editor platforms and technologies and write repetitive glue code for every integration. The Language Server Protocol (LSP) was invented to solve this problem.

LSP standardizes interactions between developer tooling (referred to as “servers” in the protocol) and text editors (“clients”) over a technology-neutral base layer. This completely decouples clients from servers meaning that you, as a tooling author, don’t have to write tons of glue code when targeting an editor that supports LSP. Plus, you can write your server in any language you like without having to worry about inter-process communication. LSP was first introduced in Visual Studio Code but it’s supported by a decent number of editors by now.

How does LSP work?

LSP runs over remote procedure calls using the JSON-RPC protocol (more specifically, version two of JSON-RPC). In JSON-RPC one client and one server exchange messages that either describe a request, response, or notification. Requests can be thought of as function calls with a given set of arguments that require a response. Responses either carry a return value or an error. Notifications are similar to requests, except that they don’t require a response from the other party, so they’re completely asynchronous. An example JSON-RPC request to add two numbers could look like this:

{ "jsonrpc": "2.0", "id": 1, "method": "add", "params": { "a": 2, "b": 3 } }

Followed by a response:

{ "jsonrpc": "2.0", "id": 1, "result": 5 }

Despite using the terms “client” and “server”, the protocol is entirely bidirectional. It requires a bidirectional communication channel to transmit messages, which could be anything from WebSockets to raw TCP connections, or Unix domain sockets. The LSP specification includes recommendations for handling communication channels. In practice, many language servers use standard input and output. The specification also describes the RPC layer in more detail.

On top of JSON-RPC, LSP defines a set of standard requests and notifications for servers to expose certain features to clients. The list of requests is quite long but the specification does a good job categorizing and explaining all of them. Here are some requests that we implement in the Symflower language server:

textDocument/didSave for triggering an analysis followed by unit test generation when a source file is saved in the editor.
workspace/didChangeWorkspaceFolders for keeping track of directories that should be included in future analyses.
textDocument/codeLens and workspace/executeCommand for letting users interactively add generated unit tests to their test files in our “test review” mode.
The codeLens request returns definitions for code lenses with commands to invoke when a user interacts with them in the client (usually by clicking).
executeCommand is called when a command is then invoked (i.e. when a user actually clicks on a code lens).

Starting a language server from a VS Code extension

Continue reading on Symflower's blog:
Integrating a language server (LSP) in a VS Code extension

Why symbolic execution is the leading-edge method for generating test values

Symflower — Sun, 20 Mar 2022 23:18:33 +0000

In the first blog post of our blog series on Symflower's Core Technology, we explained how symbolic execution works and how we apply it to test source code, and generate unit tests. In this article, we will explain why we decided to go with it, and why symbolic execution is superior to other methods that aim to generate test values.

Methods for automated test value generation

When picking a technique to receive input values for different software tests, we can differentiate between white-box and black-box testing:

White-box testing takes the source code into account and picks specific test values to trigger certain paths of code to verify their behavior. If the code is exhaustively analyzed, for instance, by a clever algorithm, one aims to choose the test inputs in a manner that triggers all possible execution paths in order to reveal all the unexpected and incorrect outcomes. Examples for white-box testing techniques include bounded model checking and symbolic execution.

Black-box testing checks the correctness of programs without looking into the code, which means that the project is not analyzed. Instead, many tests are executed with any possible values. There are different methods to decide values such as boundary-value analysis and fuzzing.

To learn more about software testing itself, check out our guide to software testing.

In the following sections, we discuss each method with examples, as well as how they work, and their limitations.

Boundary-value analysis

Boundary-value analysis is commonly used by manual testers and thought for seminars on software test certifications. For generating test values with this method, the input value domains are partitioned. For instance, when testing a function that takes an integer as input, we could split the integer range every hundred steps into intervals from -100 to 0, 0 to 100, 100 to 200, and so on. Then, a minimum, maximum, and middle value are picked to test the code. Additionally, values just below and above the minimum and maximum are used. In our example, we could pick the values -1, 0, 1, 50, 99, 100, and 101 as inputs to test the function in the interval from 0 to 100. The idea behind this technique is to catch likely corner cases.

The following example is a function that takes an integer array and an integer. If the given integer has the value 123, an index access is performed on the array. This access results in an index-out-of-range exception if the array does not contain an element at the desired position 456.

func foobar(list []int, x int) int {
  if x == 123 {
    return list[456]
  }
  return 0
}

The easiest way to trigger the exception is to call the function with an empty list, or just an undefined list (in Go this is nil in Java it is null) and the number 123 for x. This is how such a test could look like, and is one of the tests Symflower generates for the given function:

package tutorial

import (
  "testing"
)

func TestSymflowerFoobar(t *testing.T) {
  list := []int{}
  x := 123

  foobar(array, x)
}

However, with boundary-value analysis, countless tests are performed, tests that are likely to still miss the index-out-of-range exception.

Since the source code is not analyzed, possible misbehavior of a program such as the index-out-of-range exception in the example above is not specifically searched for, and is only revealed with pure luck. Therefore, this method can only achieve poor coverage in real-world source code. For example, the "if" branch of the function above is only covered by boundary-value analysis if we are lucky enough to pick exactly the right value for x.

Fuzzing

With fuzzing, random values are produced automatically to test the code. The function under test is executed repeatedly with the generated input values in order to reveal problematic behavior, such as program failures.

However, finding bugs like the index-out-of-range exception in the example above in a reasonable amount of time depends on how the fuzzer chooses the input values. How long does it take the generator to produce our malicious input 123? Has the random value generation stopped before that? If so, the fuzzer does not expose the bug. Covering the input domain is even trickier when dealing with more complex structures, as the possibilities for different inputs explode. See our blog post about the Go fuzzer to view an analysis of a simple example about strings. Even though Go's native fuzzing is very powerful, the tool does not execute the desired string case of the example in the first 20 (!) minutes. Hence, similarly to boundary-value analysis, the reached coverage of fuzzing depends on luck, while wasting resources for an awful lot of unnecessary program executions.

Additionally, the generated test values are not like a human would pick them. For instance, to reveal an error path that is guarded with the condition x > 123, the assignment x = 124 in the corresponding test is much more intuitive than an arbitrary 32-bit integer greater than 123, let's say x = 274032187. As a result, developers have a hard time understanding and using tests that contain these values for debugging found problems.

Imagine a string structure "Color" that only allows the strings "red", "blue", "green" and "yellow", and the following piece of code:

switch myColorInput {
case "blue":
  return "sky"
case "green":
  return "grass"
case "yellow":
  return "sun"
default:
  return "rainbow"
}

A fuzzer may provide a test that covers the default case, but since the underlying structure is not analyzed, the chance is virtually zero that a value "red" is generated. Consequently, it is not obvious from the produced random string values to the developer that they missed one case.

Bounded model checking

Bounded model checking is a technique that takes the given source code into account, and can therefore produce higher coverage. For analyzing programs, models of the given source code are specified as logic formulas that are checked for satisfiability. Read on to learn about satisfiability problems and how to solve them. For deeper insights into bounded model checking of finite program executions, we recommend this article by Armin Biere et al..

A clear disadvantage of bounded model checking, however, is that the analyses are limited in depth and complexity, e.g. bugs that occur only on deep levels of loops and recursions cannot be found in a scaling manner or even not at all.

In the following example, the given function takes two integers and performs a number of loop iterations, which is determined by the input. If the loop is executed the right amount of times, the program fails with a division by zero. The bug may or may not be found, depending on the implementation of the bounded model checker.

func f(j int, k int) int {
  if k < 10 {
    return 0
  }

  a := 0
  for i := 0; i < j; i++ {
    if i > k {
      a++ // This needs to be executed at least 10 times to reveal the error. Hence, "j" needs to be chosen to be a high enough number.
    }
  }

  if a > k { // The edge case "a == k" was overlooked in this condition.
    return 1
  }

  return j / (a - k) // Here, the division by zero happens for "a == k".
}

Symbolic execution

With symbolic execution, the source code is executed with symbolic values instead of actual ones, meaning that the instances can be picked at the end of the analysis. Conditions on these symbolic values are collected along the way and expressed in mathematical language. Finally, values that fulfill these conditions are computed. In this way, symbolic execution is able to trigger all possible execution paths of the program with the produced input values. This technique is the premier class of program analysis since it is theoretically unlimited in depth and complexity. By utilizing symbolic execution, Symflower searches for all possible bugs in your software and supplies optimized unit tests that are easy to read and ideal for debugging. To see an example and learn more, read our blog post about symbolic execution.

Takeaways

Please note that black-box testing techniques can be helpful to check the behavior of your program. However, lots and lots of tests are generated unnecessarily, resulting in a waste of resources. And the main problem remains, as the same for bounded model checking: Not all computation paths are validated, and dangerous bugs stay hidden. This is why Symflower decided to go with symbolic execution, the champion of program analysis.

Naturally, there are also issues that are problematic for symbolic execution, such as path explosion or calls to the environment like programs that interact with files. Tackling these challenges is, however, worth it to be able to analyze arbitrary projects. Concluding, Symflower picks the best of the best methods, which is the only way of fully autonomously assuring quality for all software there is.

Stay tuned about Symflower's Core Technology and upcoming blog posts of this series. Sign up for our newsletter and follow us on Twitter, Facebook, and LinkedIn for more content.

How real-time autonomous unit testing can help you develop better software

Symflower — Sun, 20 Mar 2022 23:13:38 +0000

Having to deal with code quality issues deep in a project’s life-cycle is a shared nightmare between all developers. There is nothing worse than being knee-deep in development – or, in a worst-case scenario, already in production – and realizing that the code you’re working with is full of holes that will impact the product’s success.

When looking into catching code quality issues early on, unit testing is always an essential cornerstone. Through it, developers are capable of identifying issues and breaking points in your code in the early stages of development – however, unit tests are usually written and run after your code is already done, meaning you already lose time and effort on producing code that will most likely have to be corrected and changed due to human error either way.

But what if we told you there was another way: to have your unit tests not only autonomously generated, but also being generated real-time while you’re working on your code, available at the end of your fingertips? Let’s take a look at some of the benefits of a real-time, autonomous unit testing solution like Symflower CLI.

Real-time unit testing fits seamlessly into existing development processes

Standard testing practices usually have a set order to them: you work on your code, test it, and then work on the issues. But why would you wait until your code is completely done and compiled to see whether it works?

One of the biggest benefits of real-time autonomous unit testing is that it fits into your already existing development processes without you having to change anything. There is no need to plan for writing tests, as they are automatically generated. There are no rounds of review needed to be scheduled after the fact, as the tests are generated on-the-go as you type.

With your tests being instantly available as you work on your code, you’re able to introduce an additional layer of reactiveness and flexibility into your development processes that is not only agile at the core, but also helps you spend less time testing and debugging, reducing the overall work needed for a project to succeed.

Shift left with real-time feedback: faster iteration and error correction

Time is always of the essence when it comes to development – you want to avoid having to go back to your code at a later time just to fix problems you could have during the early stages of development. What is better than being able to do so when the code is still fresh out of your mind, and you’re aware of all its little quirks?

Automated unit testing helps people generate and run in-depth tests for all kinds of use cases, and allows for longer and more complex tests that are sometimes avoided when it comes to manual testing, mainly due to them having to be constantly monitored. By default, automation already helps with filtering many of the human errors that can happen during manual testing, but when it comes to real-time feedback, it gets even better.

With a tool like Symflower CLI generating unit tests for you right as you code, you’re not only able to identify the problems you’d eventually have to fix, but actually fix them right away as well. This creates a workflow that is more efficient and fluid than having to go back and check things after the fact. Check out how it works here:

Better test coverage and efficiency, more specific results

Automated tests are already a step up from simply doing manual testing: in fact, using automated testing alone has been proven to decrease defects in code by 20-30% and increase code quality in general.

Real-time autonomous unit testing, however, generates test cases based on the code that you’re writing as you’re writing it, which means it analyzes it bit-by-bit as it gets put down. This way of analyzing code and creating reactive test cases is able to provide for better code coverage with no dark spots, and more specific test results that can help you isolate problems easier and more granularly.

This, combined with the symbolic execution test generation method used by Symflower CLI results in a solution that can not only cover relevant execution paths and problems through dedicated unit tests, but can also do so in a more reactive and efficient way than regular unit testing.

If you want to know more about why symbolic execution is the way to go over other test generation methods, check out our article on why we decided to go with it, and why we think you should give it a go, too.

Saving time and money through failing fast

When considering automated testing, "failing fast" is key: isolating and identifying problems within the code as soon as possible helps developers save time that would be spent on working on correcting them later on, and possibly reworking parts of the code that relied on the problematic ones.

Real-time test generation takes "failing fast" to a new level, as detecting problems in the code happens instantly, and follows the code as it gets written. Finding these bugs at an earlier stage does not only save more development time than regular automated testing, but also lets developers save significant work hours, and focus on developing what’s really important: the product itself.

Having a better code quality and finding and iterating on problems faster is not only an advantage for developers, of course: through employing a real-time unit testing solution, companies can avoid the fallout that would result from a product of bad quality, as well as shorten their time to market significantly.

Convinced that real-time, autonomous unit testing is the way to go? Check out Symflower CLI, and get started today for free. For more content about software testing, sign up for our newsletter and follow us on Twitter, Facebook, and LinkedIn. See you next time!

Better table-driven tests: generating Symflower-style unit tests

Symflower — Sun, 20 Mar 2022 22:55:28 +0000

Ever wondered what the best style for writing unit tests is? We at Symflower have a certain take on that. We developed a test style optimized towards debuggability, readability and understandability. This reduces the time spent on finding failing tests and understanding test cases, and therefore allows us to spend more time on actual development. We also created a Visual Studio Code extension to help us write those unit tests. Let's take a look at both of them!

Symflower-style table-driven unit tests

In the beginning of Symflower, we were using the standard Go style for writing our table-driven tests. Over time we got increasingly unhappy with that style. The main reason was that it took some time and effort to figure out which test case was failing when we looked at the go test output and its stack traces. We, therefore, developed our own style which solves this problem. By encapsulating the test cases in a call to a validation function, we were able to get meaningful stack traces for failing tests while still sticking to a table-driven style.

Characteristics

Table-driven
Validation function
Named test cases
Structured test cases

Benefits

Meaningful stack traces for failing tests
Improved readability and understandability by named and structured test cases
Reuse of validation code

The main purpose of our style is an improvement in debuggability and readability. Both aspects help us spend less time on debugging tests and stay focused on actual coding.

Stack traces for failing tests. This is probably the most important benefit of this style. Because of the calls to the validation function, each stack trace holds a location to the failing test case. In comparison, standard Go table-driven style can only have stack traces pointing to the validation, since the test cases do not forward their source code location.
Developers often work around this problem by adding the name of the test cases to the assert messages, which can be can be skipped with the Symflower table-driven style as well.

Named test cases. By adding human-readable text, the experience for the next person who is looking at the tests is improved. The named test cases also describe the behavior the test is supposed to check. This introduces a flavor of behavior-driven development to our unit tests.
Another aspect of named tests is that they are easier to find if you are looking for a specific one.

Structured test cases. The test case definitions are split into name, input, and output by adding blank lines in between them. This way, it's always clear what the input and the expected output is. We do this regardless of whether there are multiple arguments or just a single one, it's a convention.
Another important point is that we always make use of the field names in struct literals. Otherwise, it would become unreadable when several fields are involved.

Other test styles

There are, of course, other frequently used test styles out there. We'll briefly showcase them for you, so that you have a proper base for comparison with the Symflower style.

Function per test case

This is the most basic test style. Since every test case gets its own test function, there is a lot of redundant code, and the specific behavior a test case should check is also not described.

Disadvantages

Redundant validation code
Overview of existing tests is not great
Long and hard to read test function names

Standard Go style

This is the style you see most often in Go projects. It's already table-driven, which is a huge improvement to the function-per-test-case style, since it reduces the amount of redundant code. However, there are still disadvantages, which are overcome by our style.

Disadvantages

No relevant stack traces for failing tests
No description of the behavior the test cases should check
Missing field names make tests often hard to follow
More fields and data make these tests often very hard to read and maintain

The Symflower VS Code Extension

Creating unit tests will always involve writing some boiler-plate code. That's where editor extensions come in handy. The Symflower Visual Studio Code extension allows you to generate tests in the above unit test style, and even maintain them. Please drop us a line on our public issue tracker if you are missing a feature, found a bug or just want to tell us how you like the extension and the unit test style.

Our main area of expertise at Symflower is, however, generating actual unit tests, and not just test skeletons. With this in mind, the next feature in our pipeline is going to be aimed at providing you with unit tests that are integrated in your existing test suite.
Sign up for our newsletter and follow us on Twitter, Facebook, and LinkedIn to get notified about the upcoming releases and news about programming and tech.

A COMPLETE guide on how to make Docker images even smaller

Symflower — Wed, 16 Mar 2022 09:46:36 +0000

These days everything is either moving or already running in containers when it comes to software infrastructure. The very first container of Symflower was not for our product, some service or our website, it was our continuous integration (CI). Starting off with an almost tiny 100MB, we noticed that our baby Docker container image has grown to a full-size monster with a whopping 6.2GB. Here is a step-by-step guide on how you can reduce the size of your container images and how we slimmed down our oversized monster image.

{{< info-block >}}
Please note that this article is not just a bunch of tips, but is really meant as a complete guide with debugging information on how to slim down your container images. While only Docker is mentioned, these steps and techniques are applicable to most solutions. In any case, if you find a step that we missed towards a smaller image size: tell us! We will extend the article so the next person benefits from your feedback.
{{</ info-block >}}

{{< table-of-contents >}}

As with everything in the world of software development and infrastructure, we can rely on best practices that others have already established by most likely wading through hours of painful research.

For a comprehensive guide to reducing the size of Docker images, continue reading on Symflower's blog:
A COMPLETE guide on how to make Docker images even smaller

Getting started with Visual Studio Code extension development

Symflower — Wed, 16 Mar 2022 08:31:35 +0000

Most developers have used Visual Studio Code at some point in their careers. It has been voted the most popular IDE of 2021 by respondents of the StackOverflow developer survey, and its appeal is clear as ever. While the base program provides a framework for an IDE, all of the language support and special features are delivered as extensions, which makes it easy to extend and customize your installation. So inevitably, there comes a point where you want to write your own extension. This guide will walk you through the basics of getting started with Visual Studio Code extension development.

What is a VS Code extension?

Before we jump into coding, we should clear up what a VS Code extension is on a technical level. Extensions are basically programs, written in JavaScript or TypeScript, which hook into various parts of VS Code. They provide functions for VS Code to call when certain events happen, and can programmatically interact with (some parts of) VS Code in those functions.

Extensions are distributed as ZIP files with a specific file and folder structure inside. The files contained in this structure are usually very verbose and not friendly for humans to read or write so there’s an official build tool to generate such ZIP files from source code: vsce. Its usage will be explained later on in this post.

Development is best done in VS Code itself. It supports TypeScript out of the box and comes with special tools to run and debug your extension in another instance. In principle, other editors would work as well, but you should have VS Code ready for running and testing your extension either way.

Getting started with VS Code extensions

For starters, let’s install some command-line tools for development:



npm install --global yo generator-code vsce

…and set up our project.



$ yo code

     _-----_     ╭──────────────────────────╮
    |       |    │   Welcome to the Visual  │
    |--(o)--|    │   Studio Code Extension  │
   `---------´   │        generator!        │
    ( _´U`_ )    ╰──────────────────────────╯
    /___A___\   /
     |  ~  |
   __'.___.'__
 ´   `  |° ´ Y `

? What type of extension do you want to create? New Extension (TypeScript)
? What's the name of your extension? hello-world
? What's the identifier of your extension? hello-world
? What's the description of your extension?
? Initialize a git repository? Yes
? Bundle the source code with webpack? No
? Which package manager to use? npm

Writing in /src/hello-world...
[...]

Choose “New Extension (TypeScript)” and enter your extension’s details. You can always change these settings later. Optionally, initialize a Git repository and accept the default of “No” for “Bundle the source code with webpack?”. Select the package manager on your system (most likely “npm”). After that, open the newly created folder in your editor of choice and open src/extension.ts.

This is the entry point of your extension. VS Code will evaluate this file when loading your extension — but make sure you don’t put your initialization code directly in the top-level scope of the script!

A special function called activate is intended for setup code, and is called by VS Code when an extension is first “needed” after being deactivated, freshly installed, or after VS Code was started. “Needed” in this case means that one of several Activation Events has been triggered. The generated example code demonstrates this with a command Activation Event, but we’ll also explore another way to start your extension later.

Running a VS Code extension in development mode

Let’s have a look at the generated demo code in action! As mentioned before, it registers a command which can be run in the command launcher (Ctrl+Shift+P by default), so let’s try that now.

If you’re already in VS Code, go to the “Run & Debug” tab in the leftmost sidebar. Select the “Run Extension” launch configuration in the dropdown next to the green “Run” button. Then press the “Run” button (or F5).

If you’re not working from VS Code, run



code --extensionDevelopmentPath=$PWD

…from your shell. Note that the path given to --extensionDevelopmentPath has to be absolute.

VS Code will open, either with no workspace folder at all or with a recently opened workspace. Next, just press Ctrl+Shift+P and type “hello world”. A new command called “Hello World” should show up. Select it, hit Enter and a notification should appear.

Checking back with the code, we can clearly see how this is implemented. The call to registerCommand tells VS Code what to do when the “Hello World” command is executed. However, this just provides the implementation. The definition of our command lives in the package.json file, under the contributes section.



"contributes": {
  "commands": [
    {
      "command": "hello-world.helloWorld",
      "title": "Hello World"
    }
  ]
},

A lot of extension functionality is defined in contributes: language support, settings, commands and more. These definitions are referred to as “Contribution Points”.

Back in extension.ts, we can see that the return value from registerCommand is pushed onto context.subscriptions. What’s that all about?
“Subscriptions” might be a bit misleading here. More commonly, VS Code uses the term “Disposable”. Let’s check the docs.

“Represents a type which can release resources, such as event listening or a timer.”

Okay cool. TL;DR: most of the time, Disposables represent something that can be “stopped” or canceled (for example, providing a function to call when a command is invoked, as shown in the demo code). When your extension deactivates, context.subscriptions calls dispose on the Disposables pushed onto it, which makes it a handy tool for managing lifetime-scoped Disposables (like command handlers).

Exploring the VS Code extension API

Time to add some features. Let’s display a notification when a file is saved. It’s pretty simple: We just have to register an event listener. Since the event is related to workspaces (think editors and files), we find its handle in vscode.workspaces. onDidSaveTextDocument seems appropriate, so let’s just call it from inside the activate function:



disposable = vscode.workspace.onDidSaveTextDocument((evt) => {
  vscode.window.showInformationMessage(`Saved ${evt.fileName}`);
});

context.subscriptions.push(disposable);

Since the event listener — much like a command handler — is a “continuous thing” that can be “stopped” the registration function returns a Disposable which we have to handle. Pushing it into context.subscriptions is a good fit here since we never want to stop listening for save events while our extension is active.

Alright, let’s run that. Just press F5 to launch the last configuration again, open a text document, save and… oh no. Nothing’s happening! The problem is an easy one: our extension has not been activated yet. Remember Activation Events? As mentioned before, our extension is currently only command-activated. If you run the “Hello World” command, then try saving again, a notification should appear as expected.

We can see the configuration responsible for that in the package.json file under activationEvents.



"activationEvents": [
  "onCommand:hello-world.helloWorld"
],

Currently, only one Activation Event is registered called onCommand:hello-world.helloWorld. This event fires when the “Hello World” command is executed. Since we would like to listen to all file save events without first having to run a command, let’s replace the whole onCommand[…] string with onStartupFinished, which fires right after VS Code has started.



"activationEvents": [
  "onStartupFinished"
],

In general, you should aim for more specific Activation Events. Less extensions to start at once makes VS Code start up faster.

Now, let’s restart our launch configuration, open a file in the development host and save. Our extension finally displays a notification! By the way, if you leave the “Extension Development” instance of VS Code open while making changes, you can also press Ctrl+R to reload the window and try your changes instantly.

Let’s add a status bar item. TL;DRtD (too long, didn’t read the docs) this is the code:



disposable = vscode.window.setStatusBarMessage('Never saved anything');
context.subscriptions.push(disposable);

disposable = vscode.workspace.onDidSaveTextDocument((evt) => {
  const disposable = vscode.window.setStatusBarMessage(`Saved ${evt.fileName} at ${Date.now()}`);
  context.subscriptions.push(disposable);
});

context.subscriptions.push(disposable);

Just replace what we added for onDidSaveTextDocument before.

The status bar is part of the window, so we find its functionality in vscode.window. Makes sense! Status bar items are Disposables. Why? If you think about it: Status bar items can disappear, so it makes sense to use the Disposable interface here. We’ll just handle them via context.subscriptions again.

One thing to note from the docs:

Note that status bar messages stack and that they must be disposed when no longer used.

They stack? Well, if we add a timeout to the “saved” status bar messages only, we can see this in action. Just pass a number as the second parameter to the call.



vscode.window.setStatusBarMessage(`Saved ${evt.fileName} at ${Date.now()}`, 1000);

“Saved” messages will disappear after one second to reveal the message below (down to “Never saved anything”). This function pushes status bar messages onto a stack.

Building and installing a VS Code extension

Okay, that was enough about development workflows and general concepts. Let’s finally build that special ZIP file mentioned in the beginning so you can actually install and use your extension. Open your extension’s source directory in a terminal and run vsce package.



vsce package
Executing prepublish script 'npm run vscode:prepublish'...

> hello-world@0.0.1 vscode:prepublish /src/hello-world
> npm run compile


> hello-world@0.0.1 compile /src/hello-world
> tsc -p ./

ERROR  Make sure to edit the README.md file before you package or publish your extension.

…

Okay, apparently vsce thinks we intended to publish the extension and forgot to change the default generated README. Thanks. I like to resolve this situation with a quick echo this is not useful > README.md but you’re welcome to write a more useful README.

After this, we just re-run vsce package. This command will also display some actually helpful warnings (which you can just ignore and continue anyways). Afterwards, you get a *.vsix file. That’s the special ZIP file we mentioned, which you can open with a ZIP archive browser to explore its contents.

Installing it into your main copy of VS Code is also pretty easy: On the command line, run code --install-extension ./hello-world-0.0.1.vsix. In the GUI, go to “Extensions” and click the three horizontal dots at the top of the left sidebar. Click “Install from VSIX…” and select your VSIX file.

And that’s it! You’re now a Visual Studio Code extension author. For more in-depth information about developing extensions and API references, check out the official docs. For more VS Code guides from us, subscribe to our newsletter, and follow us on Twitter, Facebook, and LinkedIn. In the next blog posts we talk about publishing on the Marketplace, testing and our own super-advanced testing tools for VS Code extensions. Stay tuned!