Category Archives: Robotics

Learning robot simulators

June 30, 2023 09:58 ,

Grant W. Woodford, Mathys C. du Plessis, Bootstrapped Neuro-Simulation for complex robots, . Robotics and Autonomous Systems, Volume 136, 2021 DOI: 10.1016/j.robot.2020.103708.

Robotic simulators are often used to speed up the Evolutionary Robotics (ER) process. Most simulation approaches are based on physics modelling. However, physics-based simulators can become complex to develop and require prior knowledge of the robotic system. Robotics simulators can be constructed using Machine Learning techniques, such as Artificial Neural Networks (ANNs). ANN-based simulator development usually requires a lengthy behavioural data collection period before the simulator can be trained and used to evaluate controllers during the ER process. The Bootstrapped Neuro-Simulation (BNS) approach can be used to simultaneously collect behavioural data, train an ANN-based simulator and evolve controllers for a particular robotic problem. This paper investigates proposed improvements to the BNS approach and demonstrates the viability of the approach by optimising gait controllers for a Hexapod and Snake robot platform.

Posted in: Robot models , Tagged: , ,

Mixing Monte-Carlo Tree Search with Q-learning for robot learning

June 30, 2023 09:48 ,

Francesco Riccio, Roberto Capobianco, Daniele Nardi, LoOP: Iterative learning for optimistic planning on robots, . Robotics and Autonomous Systems, Volume 36, 2021 DOI: 10.1016/j.robot.2020.103693.

Efficient robotic behaviors require robustness and adaptation to dynamic changes of the environment, whose characteristics rapidly vary during robot operation. To generate effective robot action policies, planning and learning techniques have shown the most promising results. However, if considered individually, they present different limitations. Planning techniques lack generalization among similar states and require experts to define behavioral routines at different levels of abstraction. Conversely, learning methods usually require a considerable number of training samples and iterations of the algorithm. To overcome these issues, and to efficiently generate robot behaviors, we introduce LoOP, an iterative learning algorithm for optimistic planning that combines state-of-the-art planning and learning techniques to generate action policies. The main contribution of LoOP is the combination of Monte-Carlo Search Planning and Q-learning, which enables focused exploration during policy refinement in different robotic applications. We demonstrate the robustness and flexibility of LoOP in various domains and multiple robotic platforms, by validating the proposed approach with an extensive experimental evaluation.

Posted in: Applications of reinforcement learning to robots , Tagged: , ,

Deep learning RL methods for robot navigation

June 29, 2023 10:47 ,

Luong, M., Pham, C., Incremental Learning for Autonomous Navigation of Mobile Robots based on Deep Reinforcement Learning, . J Intell Robot Syst 101, 1 (2021) DOI: 10.1007/s10846-020-01262-5.

This paper presents an incremental learning method and system for autonomous robot navigation. The range finder laser sensor and online deep reinforcement learning are utilized for generating the navigation policy, which is effective for avoiding obstacles along the robot’s trajectories as well as for robot’s reaching the destination. An empirical experiment is conducted under simulation and real-world settings. Under the simulation environment, the results show that the proposed method can generate a highly effective navigation policy (more than 90% accuracy) after only 150k training iterations. Moreover, our system has slightly outperformed deep-Q, while having considerably surpassed Proximal Policy Optimization, two recent state-of-the art robot navigation systems. Finally, two experiments are performed to demonstrate the feasibility and effectiveness of our robot’s proposed navigation system in real-time under real-world settings.

Posted in: Applications of reinforcement learning to robots , Tagged: ,

Qualitative modelling of quadcopters that is claimed to be better than reinforcement learning

June 29, 2023 10:31 ,

Šoberl, D., Bratko, I. & Žabkar, Learning to Control a Quadcopter Qualitatively., . J Intell Robot Syst 100, 1097–1110 (2020) DOI: 10.1007/s10846-020-01228-7.

Qualitative modeling allows autonomous agents to learn comprehensible control models, formulated in a way that is close to human intuition. By abstracting away certain numerical information, qualitative models can provide better insights into operating principles of a dynamic system in comparison to traditional numerical models. We show that qualitative models, learned from numerical traces, contain enough information to allow motion planning and path following. We demonstrate our methods on the task of flying a quadcopter. A qualitative control model is learned through motor babbling. Training is significantly faster than training times reported in papers using reinforcement learning with similar quadcopter experiments. A qualitative collision-free trajectory is computed by means of qualitative simulation, and executed reactively while dynamically adapting to numerical characteristics of the system. Experiments have been conducted and assessed in the V-REP robotic simulator.

Posted in: Robot motion planning , Tagged: , ,

Using abstraction of dimensions in RRT motion planning

June 29, 2023 10:22 ,

Xanthidis, M., Esposito, J.M., Rekleitis, I. et al., Motion Planning by Sampling in Subspaces of Progressively Increasing Dimension, . J Intell Robot Syst 100, 777–789 (2020) DOI: 10.1007/s10846-020-01217-w.

This paper introduces an enhancement to traditional sampling-based planners, resulting in efficiency increases for high-dimensional holonomic systems such as hyper-redundant manipulators, snake-like robots, and humanoids. Despite the performance advantages of modern sampling-based motion planners, solving high dimensional planning problems in near real-time remains a considerable challenge. The proposed enhancement to popular sampling-based planning algorithms is aimed at circumventing the exponential dependence on dimensionality, by progressively exploring lower dimensional volumes of the configuration space. Extensive experiments comparing the enhanced and traditional version of RRT, RRT-Connect, and Bidirectional T-RRT on both a planar hyper-redundant manipulator and the Baxter humanoid robot show significant acceleration, up to two orders of magnitude, on computing a solution. We also explore important implementation issues in the sampling process and discuss the limitations of this method.

Posted in: Robot motion planning , Tagged: , ,

A very detailed study of the performance of propellers

June 28, 2023 12:04 ,

Scanavino, M., Vilardi, A. & Guglieri, G. An Experimental Analysis on Propeller Performance in a Climate-controlled Facility. J Intell Robot Syst 100, 505–517 (2020) DOI: 10.1007/s10846-019-01132-9.

Despite many commercial applications make extensive use of Unmanned Aircraft Systems (UAS), there is still lack of published data about their performance under unconventional weather conditions. In the last years, multirotors and fixed wing vehicles, commonly referred to as drones, have been studied in wind environments so that stability and controllability have been improved. However, other important weather variables have impact on UAS performance and they should be properly investigated for a deeper understanding of such vehicles. The primary objective of our study is the preliminary characterization of a propeller in a climate-controlled chamber. Mechanical and electrical data have been measured while testing the propeller at low pressure and cold temperatures. Test results point out that thrust and electric power are strongly affected by air density. A comparison between the experimental data and the results of the Blade Element Theory is carried out to assess the theory capability to estimate thrust in unconventional environments. The overlap between experimental data and theory computation is appropriate despite geometrical uncertainties and corroborate the need of a reliable aerodynamic database. Propeller performance data under unconventional atmospheres will be leveraged to improve UAS design, propulsion system modelling as well as provide guidelines to certify operations in extreme environments.

Posted in: Robotics , Tagged: ,

Combination of analytical models with NN learning for predicting action effects

June 28, 2023 11:47 ,

Kloss A, Schaal S, Bohg J. , Combining learned and analytical models for predicting action effects from sensory data . The International Journal of Robotics Research. 2022;41(8):778-797 DOI: 10.1177/0278364920954896.

One of the most basic skills a robot should possess is predicting the effect of physical interactions with objects in the environment. This enables optimal action selection to reach a certain goal state. Traditionally, dynamics are approximated by physics-based analytical models. These models rely on specific state representations that may be hard to obtain from raw sensory data, especially if no knowledge of the object shape is assumed. More recently, we have seen learning approaches that can predict the effect of complex physical interactions directly from sensory input. It is, however, an open question how far these models generalize beyond their training data. In this work, we investigate the advantages and limitations of neural-network-based learning approaches for predicting the effects of actions based on sensory input and show how analytical and learned models can be combined to leverage the best of both worlds. As physical interaction task, we use planar pushing, for which there exists a well-known analytical model and a large real-world dataset. We propose the use of a convolutional neural network to convert raw depth images or organized point clouds into a suitable representation for the analytical model and compare this approach with using neural networks for both, perception and prediction. A systematic evaluation of the proposed approach on a very large real-world dataset shows two main advantages of the hybrid architecture. Compared with a pure neural network, it significantly (i) reduces required training data and (ii) improves generalization to novel physical interaction.

Posted in: Developmental robotics , Tagged: ,

A so-called universal approach for modelling and controlling robots

June 28, 2023 10:25 ,

Tarokh, M., A unified kinematics modeling, optimization and control of universal robots: from serial and parallel manipulators to walking, rolling and hybrid robots, . Auton Robot 44, 1233–1248 (2020) DOI: 10.1007/s10514-020-09929-6.

The paper develops a unified kinematics modeling, optimization and control that is applicable to a wide range of autonomous and non-autonomous robots. These include hybrid robots that combine two or more modes of operations, such as combination of walking and rolling, or rolling and manipulation, as well as parallel robots in various configurations. The equations of motion are derived in compact forms that embed an optimization criterion. These equations are used to obtain various useful forms of the robot kinematics such as recursive, body and limb-end kinematic forms. Using the modeling, actuation and control equations are derived that ensure traversing a desired path while maintaining balanced operations and tip-over avoidance. Various simulation results are provided for a hybrid rolling-walking robot, which demonstrate the capabilities and effectiveness of the developed methodologies.

Posted in: Robot models , Tagged: , ,

Improving the realism of a simulator through deep learning

June 28, 2023 10:21 ,

Allevato, A.D., Schaertl Short, E., Pryor, M. et al. , Iterative residual tuning for system identification and sim-to-real robot learning, . Auton Robot 44, 1167–1182 (2020) DOI: 10.1007/s10514-020-09925-w.

Robots are increasingly learning complex skills in simulation, increasing the need for realistic simulation environments. Existing techniques for approximating real-world physics with a simulation require extensive observation data and/or thousands of simulation samples. This paper presents iterative residual tuning (IRT), a deep learning system identification technique that modifies a simulator’s parameters to better match reality using minimal real-world observations. IRT learns to estimate the parameter difference between two parameterized models, allowing repeated iterations to converge on the true parameters similarly to gradient descent. In this paper, we develop and analyze IRT in depth, including its similarities and differences with gradient descent. Our IRT implementation, TuneNet, is pre-trained via supervised learning over an auto-generated simulated dataset. We show that TuneNet can perform rapid, efficient system identification even when the true parameter values lie well outside those in the network’s training data, and can also learn real-world parameter values from visual data. We apply TuneNet to a sim-to-real task transfer experiment, allowing a robot to perform a dynamic manipulation task with a new object after a single observation.

Posted in: Robot models , Tagged: ,

Improving the simulation-to-real transfer of learning robotic skills by learning smaller skills and how to connect them in reality

June 26, 2023 12:25 ,

Julian RC, Heiden E, He Z, et al., Scaling simulation-to-real transfer by learning a latent space of robot skills, . The International Journal of Robotics Research. 2020;39(10-11):1259-1278 DOI: 10.1177/0278364920944474.

We present a strategy for simulation-to-real transfer, which builds on recent advances in robot skill decomposition. Rather than focusing on minimizing the simulation–reality gap, we propose a method for increasing the sample efficiency and robustness of existing simulation-to-real approaches which exploits hierarchy and online adaptation. Instead of learning a unique policy for each desired robotic task, we learn a diverse set of skills and their variations, and embed those skill variations in a continuously parameterized space. We then interpolate, search, and plan in this space to find a transferable policy which solves more complex, high-level tasks by combining low-level skills and their variations. In this work, we first characterize the behavior of this learned skill space, by experimenting with several techniques for composing pre-learned latent skills. We then discuss an algorithm which allows our method to perform long-horizon tasks never seen in simulation, by intelligently sequencing short-horizon latent skills. Our algorithm adapts to unseen tasks online by repeatedly choosing new skills from the latent space, using live sensor data and simulation to predict which latent skill will perform best next in the real world. Importantly, our method learns to control a real robot in joint-space to achieve these high-level tasks with little or no on-robot time, despite the fact that the low-level policies may not be perfectly transferable from simulation to real, and that the low-level skills were not trained on any examples of high-level tasks. In addition to our results indicating a lower sample complexity for families of tasks, we believe that our method provides a promising template for combining learning-based methods with proven classical robotics algorithms such as model-predictive control.

Posted in: Applications of reinforcement learning to robots , Tagged: ,