DEV Community: Matthew Smith

Open-source tool enables fuzz testing in JUnit

Matthew Smith — Fri, 02 Dec 2022 09:39:33 +0000

This week, Code Intelligence released CI Fuzz CLI, an open-source tool for Java that enables automated fuzz testing within JUnit. In this walkthrough, I want to demonstrate how you can use it to fuzz Java code using literally three commands.

The three commands:



# Initialize fuzzing
$ cifuzz init

# Create your first fuzz test
$ cifuzz create my_fuzz_test

# Run fuzz test and find bugs
$ cifuzz run my_fuzz_test

CI Fuzz CLI is an easy-to-use fuzzing tool that enables you to integrate and run fuzz tests directly from your command line. Fuzz testing is a great add-on to classic unit testing, as it adds an element of negative testing, by feeding software with millions of unexpected or invalid inputs.

CI Fuzz CLI runs on Linux, macOS, and Windows and supports integrations with common build systems and IDEs. If you want to test a Java application with CI Fuzz CLI, the only prerequisites are Java JDK (e.g., OpenDesk or Zulu) and a build system of your choice.

You can download CI Fuzz CLI here or by running the installation script in GitHub.



sh -c "$(curl -fsSL https://raw.githubusercontent.com/CodeIntelligenceTesting/cifuzz/main/install.sh)"

In the README, you will find detailed instructions on how to install the tool.

Fuzzing Use Case: How to Find Bugs in a Java Library

This example library has multiple branches that are reached under different conditions. It contains a potential remote code execution, where an attacker could inject a string and overwrite the settings.



ExploreMe.java
package com.example;
public class ExploreMe {
  // Function with multiple paths that can be discovered by a fuzzer.
  public static void exploreMe(int a, int b, String c) {
      if (a >= 20000) {
          if (b >= 2000000) {
              if (b - a < 100000) {
                  // Create reflective call
                  if (c.startsWith("@")) {
                      String className = c.substring(1);
                      try {
                        Class.forName(className);
                      } catch (ClassNotFoundException ignored) {
                      }
                  }
              }
          }
       }
   }
}

Let's take a look at how to write a fuzz test with CI Fuzz CLI that triggers this remote code execution.

How to Set Up a Fuzz Test in 3 Easy Steps

To set up CI Fuzz CLI, just follow the instruction in the documentation.

How to set up CI Fuzz CLI in Maven

CI Fuzz CLI commands will interactively guide you through the needed options and give you instructions on what to do next. You can find a complete list of commands with all options and parameters by calling cifuzz command --help.

1. Initialize the Project for CI Fuzz CLI

The first step is to initialize the project you want to test with CI Fuzz CLI.

This step will vary depending on which build system you are working with, as you will need to modify relevant build files, either pom.xml for Maven projects or build.gradle for Gradle projects. Creating a fuzz test is the same for each build system. Learn more.

To initialize a Java Maven project, first run cifuzz init in the root directory of the project. This will create cifuzz.yaml in the root directory of the project, the primary configuration file for CI Fuzz CLI.

CI Fuzz CLI will also list the dependencies you need to add to your pom.xml for your project:



<dependency>
  <groupId>com.code-intelligence</groupId>
  <artifactId>jazzer-junit</artifactId>
  <version>0.13.0</version>
  <scope>test</scope>
</dependency>

<dependency>
  <groupId>org.junit.jupiter</groupId>
  <artifactId>junit-jupiter-engine</artifactId>
  <version>5.9.0</version>
  <scope>test</scope>
</dependency>

2. Create a Fuzz Test

To create a fuzz test, cifuzz create java. This will create a fuzz test stub in the current directory. You can also use the -o flag to specify a path to create the fuzz test.

You can put the fuzz test anywhere, but we recommend you keep it close to the tested code as you would a regular unit test. In the example maven project in the cifuzz repository, the fuzz test is created in src/test/java/com/example/FuzzTestCase.java.

Here is that fuzz test:



FuzzTestCase.java
package com.example;
Import com.code_intelligence.jazzer.api.FuzzedDataProvider;
import com.code_intelligence.jazzer.junit.FuzzTest;
public class FuzzTestCase {
    @FuzzTest
    void myFuzzTest(FuzzedDataProvider data) {
        int a = data.consumeInt();
        int b = data.consumeInt();
        String c = data.consumeRemainingAsString();
        ExploreMe.exploreMe(a, b, c);
    }
}

If you are familiar with unit tests in Java, this should look mostly familiar. A few things to note about this fuzz test:

The fuzz test is in the same package as the target class
The @FuzzTest annotation identifies this method as a fuzztest method
The FuzzedDataProvider is part of the Jazzer API package and makes it easy to split fuzzing input into multiple parts and various data types.
The name of this fuzz test, as cifuzz recognizes it, is com.example.FuzzTestCase. So when you want to run this fuzz test, then run cifuzz run com.example.FuzzTestCase from the project directory.

3. Run the fuzz test

Start the fuzzing by executing cifuzz run FuzzTestCase. CI Fuzz CLI now tries to build the fuzz test and starts a fuzzing run.



$ cifuzz run FuzzTestCase 
[...]

Use ‘cifuzz finding <finding name>’ for details on a finding. 

💥[awesome_gnu] Security Issue: Remote Code Execution in exploreMe (com.example.ExploreMe:13)

Note: The crashing input has been copied to the seed corpus at: 
    src/test/resources/com/examples/MyClassFuzzTestInputs/awesome_gnu

It will now be used as a seed input for all runs of the fuzz test, including remote runs with artifacts created via ‘cifuzz bundle’ and regression tests. For more information, see:
    https://github.com/CodeIntelligenceTesting/cifuzz#regression-testing

Execution time: 3s
Average exec/s: 316880
Findings: 1
New seeds: 5 (total: 5)

Coverage Reporting and Debugging

What you can do now is take a closer look at the findings. cifuzz findings shows you a list of all findings and cifuzz finding [name] will show you detailed information about a particular finding, which is useful to understand and fix a bug.

Here we have the stack trace for the remote code execution:



[awesome_gnu] Security Issue: Remote Code Execution in exploreMe (com.example.ExploreMe:13)
Date: 2022-11-10 13:31:07.94532426 +0100 CET

    == Java Exception: com.code_intelligence.jazzer.api.FuzzerSecurityIssueHigh: Remote Code Execution

Unrestricted class loading based on externally controlled data may allow
remote code execution depending on available classes on the classpath.
    at jaz.Zer.<clinit>(Zer.java:54)
      at java.base/java.lang.Class.forName0(Native Method)
      at java.base/java.lang.Class.forName(Class.java:375)
      at com.example.ExploreMe.exploreMe(ExploreMe.java:13)
      at com.example.FuzzTestCase.myFuzzTest(FuzzTestCase.java:13)
    == libFuzzer crashing input ==
    MS: 0 ; base unit: 0000000000000000000000000000000000000000

0x40,0x6a,0x61,0x7a,0x2e,0x5a,0x65,0x72,0x0,0x1,0x0,0x40,0x0,0x0,0x0,0x75,
    @jaz.Zer\000\001\000@\000\000\000u
    artifact_prefix='./'; Test unit written to 
    ./crash-f29a1020fa966baaf8c2326a19a03b73f8e5a3c9
    Base64: QGphei5aZXIAAQBAAAAAdQ==

We also get a report of the CLI, how many lines and functions were found, and how many of the branches were covered.



Coverage Report
               File |  Functions Hit/Found |  Lines Hit/Found | Branches Hit/Found
  FuzzTestCase.java |         2/2 (100.0%) |     9/9 (100.0%) |       0/0 (100.0%)
src/explore_me.java |         1/1 (100.0%) |   23/23 (100.0%) |       8/8 (100.0%)
                    |                      |                  |     
                    |  Functions Hit/Found |  Lines Hit/Found | Branches Hit/Found                                      
              total |                  3/3 |            32    |                8/8

If you can, optimize your fuzzer for code coverage. The idea behind this recommendation is to enable the fuzzer to explore your application by taking as many different paths through your code as possible. This way, the fuzzer will be able to learn if a specific data structure or inputs are required to execute the code, and the more information the fuzzer gets about the structure of the code, the more effective your fuzz tests will be.

Closing Thoughts

Getting started with fuzzing can be challenging, as many open-source fuzzers require a lot of setup and knowledge to get your first fuzz test up and running. But I hope this article and CI Fuzz CLI can make it easier for you to get started with fuzzing, as it’s really one of the most effective (and fun) ways to test Java for integrity, reliability issues, and other vulnerabilities.

If you have any questions about this article or CI Fuzz CLI, please feel free to reach out to my colleague @markuszoppelt(Twitter)!

CI Fuzz CLI on Github: https://github.com/CodeIntelligenceTesting/cifuzz

Our test automation whisperer Josh will host a freely accesible live stream to demo some of the CLI's key features and to answer questions. More info at https://www.code-intelligence.com/webinar.

The Psychology Behind Late-Stage Software Testing

Matthew Smith — Tue, 29 Mar 2022 15:23:41 +0000

Suppose you had the choice of receiving 100€ today or 110€ in a year. If you are impatient like me, you would probably choose the 100€ today. But how would your decision change if you had to choose between 100€ today and 100 000€ in a year?

And what if we turned the decision around and gave you the choice of having to pay either 100€ today or 100 000€ in a year? Sounds easy? Interestingly, in software testing, many people choose the more expensive option in the future.

A bug that would cost 100€ to fix in the early stages of software development can end up costing 100 000€ if it has to be fixed in production (see rule of ten below). Thus, in theory, implementing early testing should be a no-brainer. Yet, we see many development teams that fail to do so. Behavioral psychology suggests that this tendency can be traced back to the way our brain perceives losses and gains at different points in time.

The Problem With Late-Stage Software Testing

Software bugs are inevitable. While there is some debate as to which combinations of testing approaches and tools are most effective at reducing them, most people agree that finding bugs earlier is better than finding them late. For this reason, tech leaders, such as Google and Microsoft, have been “shifting left” for years, and many software teams are following their lead.

According to the rule of ten, the costs of eliminating a bug increase by a factor of ten with each phase of the software development process it passes through undetected. These costs also contain the amount of developer time that is required to fix bugs that were committed months or years ago (often by other developers) and to fix consequential errors.

The reasons that cause some development teams to test late, despite the obvious benefits of doing the opposite, are typically quite rational. For example, when the necessary staffing or tooling for early-stage or continuous software testing isn’t available. But in addition to these rational motives, there are also irrational factors such as biases and cognitive errors that can subconsciously influence decision-makers to opt for late-stage testing.

Temporal Discounting in Software Engineering

In its essence, temporal discounting describes our inclination to discount rewards and losses that are in the future. The further they are away, the stronger we discount their value. Therefore, when presented with two choices of equal value, one in the now and one in the distant future, we are likely to choose the one in the now.

Temporal discounting is not just limited to monetary gains and losses. In a 2019 study by Fagerholm et al., software engineers were split into several groups. The groups were told that they were either working on a hypothetical 1-year, 2-year, 3-year, 4-year, 5-year or 10-year project. Next, all participants were presented with two options:

1. Implementing a new feature from the product backlog (5 person days)

2. Integrating a new library (5 person days), that would speed up the development process by N-days. Over the course of the whole project.

Next, participants were asked:

How many days (N) would the new library have to save you throughout the entire project, for you to prefer option two over option one?

The results showed that the longer the overall scope of the project, the more “person-day savings” it took for software engineers to prefer the long-term option. Thus, software engineers valued time-savings less if they were further away in the future. They discounted their value.

Source: Fagerholem et al., 2019

Temporal Discounting in Software Testing

Software testing is no stranger to cognitive biases, as we know of confirmation bias (also known as positive test bias) and inattentional blindness, just to name a few. When it comes to temporal discounting however, software testing has to be treated slightly differently from what we described above, since we are talking about discounted losses, instead of gains. Nonetheless, the principle remains the same: Although we rationally know that the costs (monetary and non-monetary) of unfixed vulnerabilities will increase the longer we delay software testing, it can be hard to grasp the full scope of these consequences, as they are diluted by the factor of time. Thus, from the perspective of today, the higher costs of bugs being discovered late seem a lot smaller than they actually are. Since the effect of temporal discounting depends on the amount of time that passes until we feel the effects of our losses, it is particularly strong in large projects that have a scope of several months or years.

To be fair, most development teams do not make up their software testing strategy out of thin air. Measures such as security audits and data-driven strategy all serve the purpose of making well-informed, rational decisions. But since decision-makers are still human, biases have to be taken into account and residual risk remains. Especially with constrained resources and tight schedules, decision-makers are sometimes forced to make compromises and to act fast. Even when there is enough time, resources, and staffing available to thoroughly plan software testing measures, temporal discounting should be kept in mind, and measures should be taken to minimize its effect.

Preventing Human Error in Software Testing

Although it might sound trivial, being aware of our biases already helps us to make better decisions, as it allows us to take a step back to think about them consciously. This activates the rational part of the brain, which is less susceptible to cognitive errors.

In software testing specifically, an effective approach to reducing cognitive errors is by implementing commitment devices. A commitment device is no more than a conscious decision that binds oneself to a strategic plan and takes away the option to delay. In software testing, opting for automated testing solutions can be such a commitment device. Once decision makers have decided to implement such a tool, developers can automatically run security tests at each pull request, throughout the entire development process. Automated security testing should come easier to decision-makers than manual early-stage testing methods, as it does not slow down the development process. In fact, automated testing solutions even speed up the whole process by eliminating bugs when they are still easy to fix.

Commit to Automated Software Testing

If you are interested in what such a testing approach might look like, you can check out the free demo app of the fully automated application security testing platform CI Fuzz. By directly integrating with any build environment, CI Fuzz empowers developers to run security tests without extensive software security expertise. In the free demo app, you can access real bug findings, code examples, and code coverage reports. You can also contact the dev team to onboard your own projects for continuous security testing. All you need to access the demo app is a working GitHub account.

What Do Software Testing and Marshmallows Have in Common?
During the famous Stanford marshmallow experiment in 1972, children were placed in a small room with a marshmallow in front of them. The children were then told that if they managed not to eat the marshmallow for 15 minutes, they would be rewarded with an additional marshmallow. Many children preferred to eat the marshmallow in front of them immediately instead of waiting for the second one. Just like the children who preferred the immediate gratification of eating a marshmallow now, over the discounted gratification of eating 2 marshmallows in 15 minutes, software testers can also prefer the gratification of unimpeded software development now, over the benefits of early testing, which only manifest themselves in the distant future.