Fuzz testing can effectively complement unit and integration tests in code-generation tasks performed by agents by providing automatically discovered counterexamples. This is especially useful for integration components that process external data, such as interactions with external APIs, where the space of possible inputs is often too large to cover manually, and where it is important to assess the client's robustness against a wide range of valid, malformed, and unexpected inputs.
By receiving failures and stack traces as feedback, the agent gets a much stronger feedback loop than with example-based tests alone. Fuzzing can uncover issues caused by malformed data, boundary values, unusual strings, missing fields, and rare combinations of otherwise valid parameters.
Research on LLM-generated code suggests that this approach is promising. Execution-based feedback has been shown to help with automated code generation and repair, while studies on differential fuzzing show that fuzzing can detect behavioral regressions in LLM-generated code that existing test suites may miss. There are also studies where fuzzing is integrated directly into agentic testing workflows.
Jazzer is a coverage-guided fuzzer for the JVM. It mutates inputs, keeps those that explore new execution paths, and reduces crashing inputs to smaller reproducible test cases.
Below are examples of two different fuzzing approaches. The first is robustness fuzzing, which checks whether the system remains stable when given unexpected or invalid input. The second is structured fuzzing, which checks correct behavior across valid but diverse inputs.
testImplementation 'com.code-intelligence:jazzer-junit:0.30.0'
...
@FuzzTest
void acceptsArbitrarySapResponse(@NotNull String source) {
when(restTemplate.exchange(
anyString(),
eq(HttpMethod.POST),
ArgumentMatchers.<HttpEntity<?>>any(),
eq(String.class)))
.thenReturn(ResponseEntity.ok(source));
var year = String.valueOf(Year.now().getValue());
var options = new Options();
options.setFilter(Map.of(
"yearFrom", year,
"yearTo", year));
assertDoesNotThrow(() -> {
for (DataFetcher.Result ignored : fetcher.fetch(options)) {
// Parsing occurs while the lazy iterator is consumed.
}
});
}
@FuzzTest
void processesValidSapResponse(
@InRange(min = 0, max = 99999999999L) long code,
@InRange(min = 2016, max = 2100) int year,
@NotNull @WithUtf8Length(min = 1, max = 32) String status,
@NotNull @WithUtf8Length(min = 1, max = 64) String supplier,
boolean hasPurchasingInfo,
boolean fixedVendor,
boolean deleted,
boolean hasDocument,
boolean mainComponents) throws JsonProcessingException {
var purchasingInfo = hasPurchasingInfo
? List.of(Map.of(
"Fixed_Vendor", fixedVendor ? "X" : "",
"Vendor_code", supplier,
"Deletion_flag", deleted ? "X" : ""))
: List.of();
var documentPlacement = hasDocument
? List.of(Map.of(
"Component_descr", mainComponents ? "Components MAIN" : "Components",
"Supplier", supplier))
: List.of();
var source = objectMapper.writeValueAsString(Map.of(
"records", List.of(Map.of(
"Code", code,
"Creation_date", year + "0408",
"Item_attributes", Map.of(
"Year", year,
"Code_status", status,
"Collection", "fuzz",
"Style_name", "fuzz"),
"Code_Purchasing_inforecords", purchasingInfo,
"Document_placement", documentPlacement))));
when(restTemplate.exchange(
anyString(),
eq(HttpMethod.POST),
ArgumentMatchers.<HttpEntity<?>>any(),
eq(String.class)))
.thenReturn(ResponseEntity.ok(source));
var options = new Options();
options.setFilter(Map.of(
"yearFrom", String.valueOf(year),
"yearTo", String.valueOf(year)));
assertDoesNotThrow(() -> {
for (DataFetcher.Result ignored : fetcher.fetch(options)) {
// Parsing occurs while the lazy iterator is consumed.
}
});
}
...
Jazzer executed 21,128 iterations, increased coverage from 119 to 124, expanded the corpus to 6 interesting inputs, and found no crashes or test invariant violations.
INFO: found LLVMFuzzerCustomMutator (0x106308c20). Disabling -len_control by default.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 3483268244
INFO: Loaded 1 modules (512 inline 8-bit counters): 512 [0x7667580000, 0x7667580200),
INFO: Loaded 1 PC tables (512 PCs): 512 [0x101362c40,0x101364c40),
INFO: 5 files found in /projects/app/.cifuzz-corpus/eu.example.ewc.app.sap.integration.fetcher.FetcherFuzzTest/processesValidResponse
INFO: 0 files found in /projects/app/build/resources/test/eu/example/sap/integration/fetcher/FetcherFuzzTestInputs/processesValidSapResponse
INFO: 1 files found in /var/folders/ds/frn5f9c14xl7dpq1m6_6v6yh0000gn/T/jazzer-java-seeds17930424486477809374
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: seed corpus: files: 6 min: 23b max: 27b total: 157b rss: 760Mb
#7 INITED cov: 119 ft: 119 corp: 3/77b exec/s: 0 rss: 760Mb
#18 REDUCE cov: 119 ft: 119 corp: 3/76b lim: 4096 exec/s: 0 rss: 760Mb L: 26/27 MS: 1 CustomCrossOver-
#19 REDUCE cov: 119 ft: 119 corp: 3/75b lim: 4096 exec/s: 0 rss: 760Mb L: 26/26 MS: 1 Custom-
#22 REDUCE cov: 122 ft: 122 corp: 4/101b lim: 4096 exec/s: 0 rss: 760Mb L: 26/26 MS: 3 CustomCrossOver-Custom-Custom-
#45 REDUCE cov: 123 ft: 123 corp: 5/127b lim: 4096 exec/s: 0 rss: 760Mb L: 26/26 MS: 3 CustomCrossOver-CustomCrossOver-CustomCrossOver-
#54 NEW cov: 124 ft: 124 corp: 6/153b lim: 4096 exec/s: 0 rss: 760Mb L: 26/26 MS: 5 Custom-CustomCrossOver-ShuffleBytes-Custom-CustomCrossOver-
#4096 pulse cov: 124 ft: 124 corp: 6/153b lim: 4096 exec/s: 2048 rss: 760Mb
#8192 pulse cov: 124 ft: 124 corp: 6/153b lim: 4096 exec/s: 1170 rss: 760Mb
#16384 pulse cov: 124 ft: 124 corp: 6/153b lim: 4096 exec/s: 496 rss: 760Mb
#21128 DONE cov: 124 ft: 124 corp: 6/153b lim: 4096 exec/s: 346 rss: 760Mb
Done 21128 runs in 61 second(s)
> Task :app:test
Test results: SUCCESS (3 tests, 3 successes, 0 failures, 0 skipped)
Top comments (0)