Category Archives: Robotics

Bayesian estimation of the model in model-based RL for robots

Senda, Kei, Hishinuma, Toru, Tani, Yurika, Approximate Bayesian reinforcement learning based on estimation of plant, Autonomous Robots 44(5), DOI: 10.1007/s10514-020-09901-4.

This study proposes an approximate parametric model-based Bayesian reinforcement learning approach for robots, based on online Bayesian estimation and online planning for an estimated model. The proposed approach is designed to learn a robotic task with a few real-world samples and to be robust against model uncertainty, within feasible computational resources. The proposed approach employs two-stage modeling, which is composed of (1) a parametric differential equation model with a few parameters based on prior knowledge such as equations of motion, and (2) a parametric model that interpolates a finite number of transition probability models for online estimation and planning. The proposed approach modifies the online Bayesian estimation to be robust against approximation errors of the parametric model to a real plant. The policy planned for the interpolating model is proven to have a form of theoretical robustness. Numerical simulation and hardware experiments of a planar peg-in-hole task demonstrate the effectiveness of the proposed approach.

Adapting the resolution of depth sensors and the location of the high-resolution area (fovea) as a possible attention mechanism in robots

Tasneem Z, Adhivarahan C, Wang D, Xie H, Dantu K, Koppal SJ., Adaptive fovea for scanning depth sensors, The International Journal of Robotics Research. 2020;39(7):837-855, DOI: 10.1177/0278364920920931.

Depth sensors have been used extensively for perception in robotics. Typically these sensors have a fixed angular resolution and field of view (FOV). This is in contrast to human perception, which involves foveating: scanning with the eyes’ highest angular resolution over regions of interest (ROIs). We build a scanning depth sensor that can control its angular resolution over the FOV. This opens up new directions for robotics research, because many algorithms in localization, mapping, exploration, and manipulation make implicit assumptions about the fixed resolution of a depth sensor, impacting latency, energy efficiency, and accuracy. Our algorithms increase resolution in ROIs either through deconvolutions or intelligent sample distribution across the FOV. The areas of high resolution in the sensor FOV act as artificial fovea and we adaptively vary the fovea locations to maximize a well-known information theoretic measure. We demonstrate novel applications such as adaptive time-of-flight (TOF) sensing, LiDAR zoom, gradient-based LiDAR sensing, and energy-efficient LiDAR scanning. As a proof of concept, we mount the sensor on a ground robot platform, showing how to reduce robot motion to obtain a desired scanning resolution. We also present a ROS wrapper for active simulation for our novel sensor in Gazebo. Finally, we provide extensive empirical analysis of all our algorithms, demonstrating trade-offs between time, resolution and stand-off distance.

Path planning by merging random sampling (RRT) with informed heuristics (A*)

Jonathan D Gammell, Timothy D Barfoot, Siddhartha S Srinivasa, Batch Informed Trees (BIT*): Informed asymptotically optimal anytime search, The International Journal of Robotics Research. 2020;39(5):543-567, DOI: 10.1177/0278364919890396.

Path planning in robotics often requires finding high-quality solutions to continuously valued and/or high-dimensional problems. These problems are challenging and most planning algorithms instead solve simplified approximations. Popular approximations include graphs and random samples, as used by informed graph-based searches and anytime sampling-based planners, respectively.

Informed graph-based searches, such as A*, traditionally use heuristics to search a priori graphs in order of potential solution quality. This makes their search efficient, but leaves their performance dependent on the chosen approximation. If the resolution of the chosen approximation is too low, then they may not find a (suitable) solution, but if it is too high, then they may take a prohibitively long time to do so.

Anytime sampling-based planners, such as RRT*, traditionally use random sampling to approximate the problem domain incrementally. This allows them to increase resolution until a suitable solution is found, but makes their search dependent on the order of approximation. Arbitrary sequences of random samples approximate the problem domain in every direction simultaneously, but may be prohibitively inefficient at containing a solution.

This article unifies and extends these two approaches to develop Batch Informed Trees (BIT*), an informed, anytime sampling-based planner. BIT* solves continuous path planning problems efficiently by using sampling and heuristics to alternately approximate and search the problem domain. Its search is ordered by potential solution quality, as in A*, and its approximation improves indefinitely with additional computational time, as in RRT*. It is shown analytically to be almost-surely asymptotically optimal and experimentally to outperform existing sampling-based planners, especially on high-dimensional planning problems.

Including the models into the state of a POMDP for learning them (using POMCPs in a robotic application)

Akinobu Hayashi, Dirk Ruiken, Tadaaki Hasegawa, Christian Goerick, Reasoning about uncertain parameters and agent behaviors through encoded experiences and belief planning, Artificial Intelligence, Volume 280, 2020 DOI: 10.1016/j.artint.2019.103228.

Robots are expected to handle increasingly complex tasks. Such tasks often include interaction with objects or collaboration with other agents. One of the key challenges for reasoning in such situations is the lack of accurate models that hinders the effectiveness of planners. We present a system for online model adaptation that continuously validates and improves models while solving tasks with a belief space planner. We employ the well known online belief planner POMCP. Particles are used to represent hypotheses about the current state and about models of the world. They are sufficient to configure a simulator to provide transition and observation models. We propose an enhanced particle reinvigoration process that leverages prior experiences encoded in a recurrent neural network (RNN). The network is trained through interaction with a large variety of object and agent parametrizations. The RNN is combined with a mixture density network (MDN) to process the current history of observations in order to propose suitable particles and models parametrizations. The proposed method also ensures that newly generated particles are consistent with the current history. These enhancements to the particle reinvigoration process help alleviate problems arising from poor sampling quality in large state spaces and enable handling of dynamics with discontinuities. The proposed approach can be applied to a variety of domains depending on what uncertainty the decision maker needs to reason about. We evaluate the approach with experiments in several domains and compare against other state-of-the-art methods. Experiments are done in a collaborative multi-agent and a single agent object manipulation domain. The experiments are performed both in simulation and on a real robot. The framework handles reasoning with uncertain agent behaviors and with unknown object and environment parametrizations well. The results show good performance and indicate that the proposed approach can improve existing state-of-the-art methods.

A possibly interesting paper on the estimation and adaptation of EKF-SLAM to actual models of the system and the noise that I have been unable to read due to its painful syntax

Yingzhong Tian, Heru Suwoyo, Wenbin Wang, Dziki Mbemba, Long Li, An AEKF-SLAM Algorithm with Recursive Noise Statistic Based on MLE and EM, Journal of Intelligent & Robotic Systems (2020) 97:339–355, DOI: 10.1007/s10846-019-01044-8.

Extended Kalman Filter (EKF) has been popularly utilized for solving Simultaneous Localization and Mapping (SLAM)
problem. Essentially, it requires the accurate system model and known noise statistic. Nevertheless, this condition can
be satisfied in simulation case. Hence, EKF has to be enhanced when it is applied in the real-application. Mainly, this
improvement is known as adaptive-based approach. In many different cases, it is indicated by some manners of estimating
for either part or full noise statistic. This paper present a proposed method based on the adaptive-based solution used for
improving classical EKF namely An Adaptive Extended Kalman Filter. Initially, the classical EKF was improved based on
Maximum Likelihood Estimation (MLE) and Expectation-Maximization (EM) Creation. It aims to equips the conventional
EKF with ability of approximating noise statistic and its covariance matrices recursively. Moreover, EKF was modified and
improved to tune the estimated values given by MLE and EM creation. Besides that, the recursive noise statistic estimators
were also estimated based on the unbiased estimation. Although it results high quality solution but it is followed with some
risks of non-positive definite matrices of the process and measurement noise statistic covariances. Thus, an addition of
Innovation Covariance Estimation (ICE) was also utilized to depress this possibilities. The proposed method is applied for
solving SLAM problem of autonomous wheeled mobile robot. Henceforth, it is termed as AEKF-SLAM Algorithm. In order
to validate the effectiveness of proposed method, some different SLAM-Based algorithm were compared and analyzed.
The different simulation has been showing that the proposed method has better stability and accuracy compared to the
conventional filter in term of Root Mean Square Error (RMSE) of Estimated Map Coordinate (EMC) and Estimated Path
Coordinate (EPC).

Application of Deep RL to person following by a robot, reducing the training effort of the network by reusing simple state situations in many artificially generated states

Pang, L., Zhang, Y., Coleman, S. et al., Efficient Hybrid-Supervised Deep Reinforcement Learning for Person Following Robot, J Intell Robot Syst 97, 299–312 (2020), DOI: 10.1007/s10846-019-01030-0.

Traditional person following robots usually need hand-crafted features and a well-designed controller to follow the assigned person. Normally it is difficult to be applied in outdoor situations due to variability and complexity of the environment. In this paper, we propose an approach in which an agent is trained by hybrid-supervised deep reinforcement learning (DRL) to perform a person following task in end-to-end manner. The approach enables the robot to learn features autonomously from monocular images and to enhance performance via robot-environment interaction. Experiments show that the proposed approach is adaptive to complex situations with significant illumination variation, object occlusion, target disappearance, pose change, and pedestrian interference. In order to speed up the training process to ensure easy application of DRL to real-world robotic follower controls, we apply an integration method through which the agent receives prior knowledge from a supervised learning (SL) policy network and reinforces its performance with a value-based or policy-based (including actor-critic method) DRL model. We also utilize an efficient data collection approach for supervised learning in the context of person following. Experimental results not only verify the robustness of the proposed DRL-based person following robot system, but also indicate how easily the robot can learn from mistakes and improve performance.

A model of the psychomotor behaviour of humans intended to be useful for integration with robots

Stephen Fox, Adrian Kotelba, Ilari Marstio, Jari Montonen, Aligning human psychomotor characteristics with robots, exoskeletons and augmented reality, Robotics and Computer-Integrated Manufacturing, Volume 63, 2020, DOI: 10.1016/j.rcim.2019.101922.

In previous production literature, the uncertainty of human behaviour has been recognized as a source of productivity, quality, and safety problems. However, fundamental reasons for the uncertainty of human behavior have received little analysis in the production literature. Furthermore, potential for these fundamental reasons to be aligned with production technologies in order to improve production performance has not been addressed. By contrast, in this paper, fundamental reasons for the uncertainty of human behaviour are explained through a model of psychomotor characteristics that encompasses physiology, past experiences, personality, gender, culture, emotion, reasoning, and biocybernetics. Through reference to 10 action research cases, the formal model is applied to provide guidelines for planning production work that includes robots, exoskeletons, and augmented reality.

State of the art in standards for Robotics

Z.M. Bi, Zhonghua Miao, Bin Zhang, Chris W.J. Zhang, The state of the art of testing standards for integrated robotic systems, Robotics and Computer-Integrated Manufacturing
Volume 63, June 2020, DOI: 10.1016/j.rcim.2019.101893.

Technology standards facilitate the transparency in market and the supplies of products with good quality. For manufacturers, standards make it possible to reduce the costs by mass production, and enhance system adaptabilities through integrating system modules with the standardized interfaces. However, International standards on industrial robots such as ISO-9283 were developed in 1998, and they have not updated since then. Due to every-increasing applications of robots in complex systems, there is an emerging need to advance existing standards on robots for a broader scope of system components and system integration. This paper gives an introduction of the endeavors by National Institute of Standards and Technology (NIST); especially, it overviews the recent progresses on the standardized tests of robotic systems and components. The presented work aims to identify the limitations of existing industrial standards and clarify the trend of technology standardizations for industrial robotic systems.

Symbol grounding through neural networks

Shridhar M, Mittal D, Hsu D., INGRESS: Interactive visual grounding of referring expressions, The International Journal of Robotics Research. January 2020, DOI: 10.1177/0278364919897133.

This article presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects. The key question here is to ground referring expressions: understand expressions about objects and their relationships from image and natural language inputs. INGRESS allows unconstrained object categories and rich language expressions. Further, it asks questions to clarify ambiguous referring expressions interactively. To achieve these, we take the approach of grounding by generation and propose a two-stage neural-network model for grounding. The first stage uses a neural network to generate visual descriptions of objects, compares them with the input language expressions, and identifies a set of candidate objects. The second stage uses another neural network to examine all pairwise relations between the candidates and infers the most likely referred objects. The same neural networks are used for both grounding and question generation for disambiguation. Experiments show that INGRESS outperformed a state-of-the-art method on the RefCOCO dataset and in robot experiments with humans. The INGRESS source code is available at

Adapting perception to environmental changes explicitly

Sriram Siva, Hao Zhang, Robot perceptual adaptation to environment changes for long-term human teammate following, The International Journal of Robotics Research. January 2020, DOI: 10.1177/0278364919896625.

Perception is one of the several fundamental abilities required by robots, and it also poses significant challenges, especially in real-world field applications. Long-term autonomy introduces additional difficulties to robot perception, including short- and long-term changes of the robot operation environment (e.g., lighting changes). In this article, we propose an innovative human-inspired approach named robot perceptual adaptation (ROPA) that is able to calibrate perception according to the environment context, which enables perceptual adaptation in response to environmental variations. ROPA jointly performs feature learning, sensor fusion, and perception calibration under a unified regularized optimization framework. We also implement a new algorithm to solve the formulated optimization problem, which has a theoretical guarantee to converge to the optimal solution. In addition, we collect a large-scale dataset from physical robots in the field, called perceptual adaptation to environment changes (PEAC), with the aim to benchmark methods for robot adaptation to short-term and long-term, and fast and gradual lighting changes for human detection based upon different feature modalities extracted from color and depth sensors. Utilizing the PEAC dataset, we conduct extensive experiments in the application of human recognition and following in various scenarios to evaluate ROPA. Experimental results have validated that the ROPA approach obtains promising performance in terms of accuracy and efficiency, and effectively adapts robot perception to address short-term and long-term lighting changes in human detection and following applications.