DEV Community

loading...

Testing robotics systems

Michael Jae-Yoon Chung
A roboticist.
Updated on ・3 min read

Canonical URL: https://mjyc.github.io/2020/12/16/testing.html

Testing robotics systems is hard. Here are my strategies.

Unit and property-based testing

Robotics systems often consist of layers of application programming interfaces (APIs). While it is difficult to test the lowest-level API like a ROS hardware driver, but everything above can be tested by simulating responses from a lower-layer API using any unit testing framework such as unittest (python), jest (javascript), gtest (C++). For example, one could test perception algorithms like tabletop pose detector by simulating input images or point clouds via previously stored data (e.g., benchmark data) or algorithmically generated data. The testing approach of algorithmically generating data is interesting because it allows property-based testing, i.e., generating test cases^1. This requires designing (simple) environment generation algorithms but one could simply use existing algorithms like the ones available in PythonRobotics. The key ideas are (1) testing small and independent layers of APIs using (2) algorithmically generated inputs that one could also easily compute expected outputs.

^1 inspired by model-based UI testing frameworks
^2 Zhoulai Fu and Francisco Martinez Lasaca gave a talk “Experiences with Fuzz Testing ROS Component” that covered a similar approach “fuzz testing” and shared their code

Sequential property-based testing

Many robotics algorithms work with sequential data. For example, think of state estimation, control, motion planning algorithms. Out of the box, property-based testing tools do not provide creating sequential data. So the first step to testing time-dependent algorithm is updating property-based testing tools to be able to generate sequential random data. Creating a sequential sampler that does not dependent on the outputs of testing algorithms is easy (e.g., inputs for state estimation algorithms) but otherwise, one requires to create reactive environments using existing tools like combinators. At this point, basically one requires to build a tiny simulation environment. While this sounds daunting, it's actually not thanks to many existing physics simulation libraries and game programming design patterns (e.g., entity-component systems, etc.).
The second problem is verifying test cases over time. For example, one might want to check whether the outputs of a state estimation algorithm ever return null or test if a motion planning algorithm ever returns an invalid pose. I like to use linear temporal logic (LTL) to verify such temporal properties due to the LTL's simplicity. There are some subtleties of using LTLs with property-based testing tools and I'll elaborate on those in the future (if I find more time).

Physical integration testing

Like everything robotics, one must test programs with the real robots in real world at some point. One approach is integrating continuous integration testing with real robots. There was a great talk about this topic from fetch regarding this topic. A more recent talk about AWS Robomaker also touches on this topic, too.

Closing meme

Testing is important but being ultra-selective about which code to add to a repository is also important because maintenance is hard.

source: https://github.com/dominictarr/push-streams-talk

Discussion (2)

Collapse
bugrahasbek profile image
Buğra Hasbek

Hi Micheal,

I think it is a good idea to test the software in a simulation tool such as gazebosim.org/ before field testing.

Some keywords you may find interesting: SITL (software in the loop), HITL (hardware in the loop)

Have a good one!

Collapse
mjyc profile image
Michael Jae-Yoon Chung Author

Hi Buğra, thank you for sharing your suggestions & relevant keywords!

While I agree that spinning gazebo is a good idea, esp., for integration testing purposes, I'm coming from a robotics app developer's perspective where I'd like to borrow lightweight model-based testing approaches (e.g., medium.com/criteo-labs/detecting-t...) to potentially even decouple the robot application layer from the rest of the (build) system to allow rapid prototyping of behaviors. I definitely agree that all applications must be tested in a more proper simulation environment like gazbo (and with a real robot eventually) but was thinking about ways to reduce the prototyping cycle time.