Francesco Amigoni, Matteo Luperto, Viola Schiaffonati,Toward generalization of experimental results for autonomous robots, Robotics and Autonomous Systems, Volume 90, April 2017, Pages 4-14, ISSN 0921-8890, DOI: 10.1016/j.robot.2016.08.016.
In this paper we discuss some issues in the experimental evaluation of intelligent autonomous systems, focusing on systems, like autonomous robots, operating in physical environments. We argue that one of the weaknesses of current experimental practices is the low degree of generalization of experimental results, meaning that knowing the performance a robot system obtains in a test setting does not provide much information about the performance the same system could achieve in other settings. We claim that one of the main obstacles to achieve generalization of experimental results in autonomous robotics is the low degree of representativeness of the selected experimental settings. We survey and discuss the degree of representativeness of experimental settings used in a significant sample of current research and we propose some strategies to overcome the emerging limitations.