Canonical URL: https://mjyc.github.io/2020/12/16/testing.html
Testing robotics systems is hard. Here are my strategies.
^1 inspired by model-based UI testing frameworks
^2 Zhoulai Fu and Francisco Martinez Lasaca gave a talk “Experiences with Fuzz Testing ROS Component” that covered a similar approach “fuzz testing” and shared their code
Many robotics algorithms work with sequential data. For example, think of state estimation, control, motion planning algorithms. Out of the box, property-based testing tools do not provide creating sequential data. So the first step to testing time-dependent algorithm is updating property-based testing tools to be able to generate sequential random data. Creating a sequential sampler that does not dependent on the outputs of testing algorithms is easy (e.g., inputs for state estimation algorithms) but otherwise, one requires to create reactive environments using existing tools like combinators. At this point, basically one requires to build a tiny simulation environment. While this sounds daunting, it's actually not thanks to many existing physics simulation libraries and game programming design patterns (e.g., entity-component systems, etc.).
The second problem is verifying test cases over time. For example, one might want to check whether the outputs of a state estimation algorithm ever return
null or test if a motion planning algorithm ever returns an invalid pose. I like to use linear temporal logic (LTL) to verify such temporal properties due to the LTL's simplicity. There are some subtleties of using LTLs with property-based testing tools and I'll elaborate on those in the future (if I find more time).
Like everything robotics, one must test programs with the real robots in real world at some point. One approach is integrating continuous integration testing with real robots. There was a great talk about this topic from fetch regarding this topic. A more recent talk about AWS Robomaker also touches on this topic, too.
Testing is important but being ultra-selective about which code to add to a repository is also important because maintenance is hard.