I planted 40 small changes (mutations) into the MAVLink C parser (c_library_v2, commit 56f6435e, file mavlink_helpers.h) and checked whether my test setup notices them. The setup is a libFuzzer harness with AddressSanitizer in a local container. With the seed corpus it caught 22 and missed 18, no timeouts.
The 18 misses are the interesting part.
1. Most misses are not equivalent mutants. For 15 of the 18 a targeted input shows an observable difference between original and mutant. For 3 I found none after three attempts. I treat those as equivalent candidates, not as proven equivalent.
2. A crash only harness sees a small part. I sorted the 15 observable misses by the first point where the difference becomes visible (my own classification, not a measurement): 2 runtime or memory integrity, 6 protocol acceptance and parser state, 5 semantic or API behaviour, 2 outbound representation. 13 of the 15 change output without crashing. They need assertions on acceptance, decoded fields or sent bytes.
3. The detection rate depends on the corpus. Earlier runs gave 21 and 22 of 40. In the forum discussion I found that the difference comes from the directory replayed after fuzzing: the run's own corpus in the runs with 21, the 219 seed files in the runs with 22. So 22 of 40 is the figure for this corpus, not a stable property of the parser. This explanation is not yet part of the signed record.
What I am not claiming: all differences come from mutants I planted myself. They show gaps in my test setup, not defects in the released library.
Only source code and local builds, no running systems. Runs, inputs and results are hashed into a hash chain.
Where would you cut the categories differently, and which observation points do you assert in your own parser tests?
Top comments (0)