It's been 33 years since then University of Wisconsin Professor Barton Miller began fuzz testing software with his graduate students, and it would be an understatement to say both computer programs and the tools used to test them have advanced quite a bit. Long gone are the days when inputing purely random bits of code meant to induce crashes was the norm. These days nearly all software development companies are taking a more target-oriented and thorough approach to testing their products and services, and those who aren't are at a big disadvantage. In this post, I'll walk through the different fuzz testing methods and discuss how best to use them.
The first method I'll discuss is the earliest and most primitive and is known as Black Box testing. Black Box testing is a test where a program is subjected to random inputs to induce unpredictable behavior. It is an overall negative test (we wouldn't expect a chat app to regularly handle an input of 37 different special character username inputs, right?) and does not take valid or semi-valid inputs into account. This lack of focus in Black Box testing makes it fairly limited in its use, since nearly all test cases are immediately and easily discarded by the program. Additionally, reading the test results from randomized testing can be problematic and cumbersome. That said, at the outermost trust boundaries it can be viable for finding bugs in simple password barriers. It also does not require source code since it is a method that is utilized from outside of the program structure.
The next method is much more focused than Black Box and is called the Mutation Fuzz test. Mutation tests make an attempt at "tricking" the tested program by tweaking valid and semi-valid inputs. The test is "aware" of at least some of the program structure and uses existing inputs, called seeds, to creating thousands of minor variations of inputs that are deemed valid enough to not be immediately rejected by the program. This testing is not only effective at stressing edge cases and inducing strange behavior, but it also finds deeper bugs than more ham handed methods and is easy to set up and implement. It does have longer runtimes, however, which can be a big factor if testing is to be done at multiple stages of development.
The third fuzzing method I will discuss is the Generation
Fuzz method. As the name implies, Generation testing generates test with a template that is designed specifically for each trust boundary within the program. The advantages here are many; with specific tests we not only have shorter runtimes, we also have clearer results and there is an added level of context to the data we get back from the test results. Sometimes it is not only important to know what input caused a bug or crash, but under what conditions and at what point in within the program the event occurred. Generation testing also provides deeper code coverage and is generally reusable for regression testing as the program is updated. There is, of course, a simple downside to this method - expense.
The last method I will discuss is the Feedback Fuzzing method. Feedback tests are the latest in fuzzing software and involves combining the principles of Mutation testing with smart algorithms that measure code coverage as the test is run. The test then uses existing inputs and coverage data to "learn" about the structure of the program and how to effectively and efficiently stress possible vulnerabilities by slightly changing inputs. Feedback tests provide the highest degree of code coverage and exposes the deepest bugs, but runtimes can be pretty slow. It is also a tool that shouldn't be used at early stages of development since it would need to be updated frequently in tandem with program updates.
So there you have it, the four main Fuzzing methods. To date there is no one-size-fits-all fuzz test, and it is up to the developer to decide which one fits best at different stages of development. But one thing is clear; integrating fuzz testing into development is a must, especially for equipment and programs that have the highest standards of security, such as medical equipment or aircraft instrumentation. Furthermore, fuzz testing can be done my more than just software developers. Hackers and other bad actors are also using these techniques to exploit zero-day vulnerabilities so taking the same approach in an effort to the contrary is a worthwhile endeavor. And lastly, who really enjoys writing hundreds of test cases anyway?
Top comments (0)