Category Archives: Robotics

A nice summary of RL applied to robot navigation

October 13, 2023 09:20 ,

N. Khlif, N. Khraief and S. Belghith, Reinforcement Learning for Mobile Robot Navigation: An overview IEEE Information Technologies & Smart Industrial Systems (ITSIS), Paris, France, 2022, pp. 1-7 DOI: 10.1109/ITSIS56166.2022.10118362.

For several years, research shows that interest in autonomous mobile robots is increasing and it has more and more grown. Autonomous mobile robots is an object of discussion but nowadays it’s an emerging topic due to the all progress related to field like autonomous driving and UAV (drones). Integrating intelligence into robotic systems requires solving various research problems, including one of the most important problems of mobile robotic systems: navigation. Find the answers to the following three questions: What is the localisation of the robot? Where are the robot going? How can it get there? presenting the solution of mobile robot navigation problem. These questions are answered by basic navigation parts which are localization, mapping and path planning. The paper present an overview of research on autonomous mobile robot navigation. First, a quick introduction to the various features of navigation. We also discuss machine learning and reinforcement learning in mobile robotics. Furthermore, we will discuss some path planning techniques. Some future directions are also suggested.

Posted in: Applications of reinforcement learning to robots , Tagged: ,

Mixing rule-based and reinforcement learning navigation for robots

October 13, 2023 09:18 ,

Y. Zhu, Z. Wang, C. Chen and D. Dong, Rule-Based Reinforcement Learning for Efficient Robot Navigation With Space Reduction, IEEE/ASME Transactions on Mechatronics, vol. 27, no. 2, pp. 846-857, April 2022 DOI: 10.1109/TMECH.2021.3072675.

For real-world deployments, it is critical to allow robots to navigate in complex environments autonomously. Traditional methods usually maintain an internal map of the environment, and then design several simple rules, in conjunction with a localization and planning approach, to navigate through the internal map. These approaches often involve a variety of assumptions and prior knowledge. In contrast, recent reinforcement learning (RL) methods can provide a model-free, self-learning mechanism as the robot interacts with an initially unknown environment, but are expensive to deploy in real-world scenarios due to inefficient exploration. In this article, we focus on efficient navigation with the RL technique and combine the advantages of these two kinds of methods into a rule-based RL (RuRL) algorithm for reducing the sample complexity and cost of time. First, we use the rule of wall-following to generate a closed-loop trajectory. Second, we employ a reduction rule to shrink the trajectory, which in turn effectively reduces the redundant exploration space. Besides, we give the detailed theoretical guarantee that the optimal navigation path is still in the reduced space. Third, in the reduced space, we utilize the Pledge rule to guide the exploration strategy for accelerating the RL process at the early stage. Experiments conducted on real robot navigation problems in hex-grid environments demonstrate that RuRL can achieve improved navigation performance.

Posted in: Applications of reinforcement learning to robots , Tagged: ,

Incremental learning (i.e., non-stationary environments, online -live- learning, task adaptation, life-long learning,…) for robots with Q-learning

October 6, 2023 08:53 ,

Y. Hu, D. Li, Y. He and J. Han, Incremental Learning Framework for Autonomous Robots Based on Q-Learning and the Adaptive Kernel Linear Model IEEE Transactions on Cognitive and Developmental Systems, vol. 14, no. 1, pp. 64-74, March 2022 DOI: 10.1109/TCDS.2019.2962228.

The performance of autonomous robots in varying environments needs to be improved. For such incremental improvement, here we propose an incremental learning framework based on Q -learning and the adaptive kernel linear (AKL) model. The AKL model is used for storing behavioral policies that are learned by Q -learning. Both the structure and parameters of the AKL model can be trained using a novel L2-norm kernel recursive least squares (L2-KRLS) algorithm. The AKL model initially without nodes and gradually accumulates content. The proposed framework allows to learn new behaviors without forgetting the previous ones. A novel local \u03b5 -greedy policy is proposed to speed the convergence rate of Q -learning. It calculates the exploration probability of each state for generating and selecting more important training samples. The performance of our incremental learning framework was validated in two experiments. A curve-fitting example shows that the L2-KRLS-based AKL model is suitable for incremental learning. The second experiment is based on robot learning tasks. The results show that our framework can incrementally learn behaviors in varying environments. Local \u03b5 -greedy policy-based Q -learning is faster than the existing Q -learning algorithms.

Posted in: Applications of reinforcement learning to robots , Tagged: ,

Adaptation of model-free RL to variations in the task under continuous state and action spaces applied to robot grasping

October 6, 2023 08:38 ,

Shahid, A.A., Piga, D., Braghin, F. et al. Continuous control actions learning and adaptation for robotic manipulation through reinforcement learning, Auton Robot 46, 483\u2013498 (2022) DOI: 10.1007/s10514-022-10034-z.

This paper presents a learning-based method that uses simulation data to learn an object manipulation task using two model-free reinforcement learning (RL) algorithms. The learning performance is compared across on-policy and off-policy algorithms: Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC). In order to accelerate the learning process, the fine-tuning procedure is proposed that demonstrates the continuous adaptation of on-policy RL to new environments, allowing the learned policy to adapt and execute the (partially) modified task. A dense reward function is designed for the task to enable an efficient learning of the agent. A grasping task involving a Franka Emika Panda manipulator is considered as the reference task to be learned. The learned control policy is demonstrated to be generalizable across multiple object geometries and initial robot/parts configurations. The approach is finally tested on a real Franka Emika Panda robot, showing the possibility to transfer the learned behavior from simulation. Experimental results show 100% of successful grasping tasks, making the proposed approach applicable to real applications.

Posted in: Applications of reinforcement learning to robots, Industrial robots , Tagged: ,

Leveraging embodiment: finding an optimal viewpoint in the robot environment for improving scene description

September 29, 2023 15:49 ,

Tan, Sinan, Guo, Di, Liu, Huaping, Zhang, Xinyu, Sun, Fuchun Embodied scene description, Autonomous Robots 46(1) DOI: 10.1007/s10514-021-10014-9.

Embodiment is an important characteristic for all intelligent agents, while existing scene description tasks mainly focus on analyzing images passively and the semantic understanding of the scenario is separated from the interaction between the agent and the environment. In this work, we propose the Embodied Scene Description, which exploits the embodiment ability of the agent to find an optimal viewpoint in its environment for scene description tasks. A learning framework with the paradigms of imitation learning and reinforcement learning is established to teach the intelligent agent to generate corresponding sensorimotor activities. The proposed framework is tested on both the AI2Thor dataset and a real-world robotic platform for different scene description tasks, demonstrating the effectiveness and scalability of the developed method. Also, a mobile application is developed, which can be used to assist visually-impaired people to better understand their surroundings.

Posted in: Mobile robot mapping , Tagged: ,

Using physical human-robot interaction to deduce the goals of the human during learning

September 29, 2023 15:31 ,

Losey DP, Bajcsy A, O’Malley MK, Dragan AD, Physical interaction as communication: Learning robot objectives online from human corrections, The International Journal of Robotics Research. 2022;41(1):20-44 DOI: 10.1177/02783649211050958.

When a robot performs a task next to a human, physical interaction is inevitable: the human might push, pull, twist, or guide the robot. The state of the art treats these interactions as disturbances that the robot should reject or avoid. At best, these robots respond safely while the human interacts; but after the human lets go, these robots simply return to their original behavior. We recognize that physical human\u2013robot interaction (pHRI) is often intentional: the human intervenes on purpose because the robot is not doing the task correctly. In this article, we argue that when pHRI is intentional it is also informative: the robot can leverage interactions to learn how it should complete the rest of its current task even after the person lets go. We formalize pHRI as a dynamical system, where the human has in mind an objective function they want the robot to optimize, but the robot does not get direct access to the parameters of this objective: they are internal to the human. Within our proposed framework human interactions become observations about the true objective. We introduce approximations to learn from and respond to pHRI in real-time. We recognize that not all human corrections are perfect: often users interact with the robot noisily, and so we improve the efficiency of robot learning from pHRI by reducing unintended learning. Finally, we conduct simulations and user studies on a robotic manipulator to compare our proposed approach with the state of the art. Our results indicate that learning from pHRI leads to better task performance and improved human satisfaction.

Posted in: Applications of reinforcement learning to robots , Tagged: ,

A grammar for symbolic robot maps that allows for mapping unknown spaces

September 25, 2023 16:57 ,

B. Talbot, F. Dayoub, P. Corke and G. Wyeth, Robot Navigation in Unseen Spaces Using an Abstract Map, IEEE Transactions on Cognitive and Developmental Systems, vol. 13, no. 4, pp. 791-805, Dec. 2021 DOI: 10.1109/TCDS.2020.2993855.

Human navigation in built environments depends on symbolic spatial information which has unrealized potential to enhance robot navigation capabilities. Information sources, such as labels, signs, maps, planners, spoken directions, and navigational gestures communicate a wealth of spatial information to the navigators of built environments; a wealth of information that robots typically ignore. We present a robot navigation system that uses the same symbolic spatial information employed by humans to purposefully navigate in unseen built environments with a level of performance comparable to humans. The navigation system uses a novel data structure called the abstract map to imagine malleable spatial models for unseen spaces from spatial symbols. Sensorimotor perceptions from a robot are then employed to provide purposeful navigation to symbolic goal locations in the unseen environment. We show how a dynamic system can be used to create malleable spatial models for the abstract map, and provide an open-source implementation to encourage future work in the area of symbolic navigation. The symbolic navigation performance of humans and a robot is evaluated in a real-world built environment. This article concludes with a qualitative analysis of human navigation strategies, providing further insights into how the symbolic navigation capabilities of robots in unseen built environments can be improved in the future.

Posted in: Mobile robot mapping , Tagged:

A survey of morphological development in developmental robotics

September 25, 2023 16:51 ,

M. Naya-Varela, A. Faíña and R. J. Duro, Morphological Development in Robotic Learning: A Survey, IEEE Transactions on Cognitive and Developmental Systems, vol. 13, no. 4, pp. 750-768 DOI: 10.1109/TCDS.2021.3052548.

Humans and animals undergo morphological development (MD) processes from infancy to adulthood that have been shown to facilitate learning. However, most of the work on developmental robotics (DRs) considers fixed morphologies, addressing only the development of the cognitive system of the robots. This article aims to provide a survey of the work that is being carried out within the relatively new field of MD in robots. In particular, it contemplates MD as the changes that occur in the properties of the joints, links and sensors of a robot during its lifetime and focuses on the work carried out by different authors to try to determine their influence on robot learning. To this end, walking, reaching, grasping and vocalization have been identified as the four most representative tasks addressed in the field, clustering the work of the different authors around them. The approach followed is multidisciplinary, discussing the relationships among DRs, embodied artificial intelligence and developmental psychology in humans in general, as well as for each of the tasks, and providing an overview of the many avenues of research that are still open in this field.

Posted in: Developmental robotics , Tagged:

Modelling network delay in the remote estimation of the robot state for networked telerobots

September 25, 2023 11:35 ,

Barnali Das, Gordon Dobie, Delay compensated state estimation for Telepresence robot navigation, Robotics and Autonomous Systems, Volume 146, 2021 DOI: 10.1016/j.robot.2021.103890.

Telepresence robots empower human operators to navigate remote environments. However, operating and navigating the robot in an unknown environment is challenging due to delay in the communication network (e.g.,�distance, bandwidth, communication drop-outs etc.), processing delays and slow dynamics of the mobile robots resulting in time-lagged in the system. Also, erroneous sensor data measurement which is important to estimate the robot\u2019s true state (positional information) in the remote environment, often create complications and make it harder for the system to control the robot. In this paper, we propose a new approach for state estimation assuming uncertain delayed sensor measurements of a Telepresence robot during navigation. A new real world experimental model, based on Augmented State Extended Kalman Filter (AS-EKF), is proposed to estimate the true position of the Telepresence robot. The uncertainty of the delayed sensor measurements have been modelled using probabilistic density functions (PDF). The proposed model was successfully verified in our proposed experimental framework which consists of a state-of-the-art differential-drive Telepresence robot and a motion tracking multi-camera system. The results show significant improvements compared to the traditional EKF that does not consider uncertain delays in sensor measurements. The proposed model will be beneficial to build a real time predictive display by reducing the effect of visual delay to navigate the robot under the operator\u2019s control command, without waiting for delayed sensor measurements.

Posted in: Networked telerobots , Tagged: , ,

Analysis of the under-optimality of path lengths when path planning is carried out on a grid instead of the continuous world

September 15, 2023 10:15 ,

James P. Bailey, Alex Nash, Craig A. Tovey, Sven Koenig, Path-length analysis for grid-based path planning, Artificial Intelligence, Volume 301, 2021, DOI: 10.1016/j.artint.2021.103560.

In video games and robotics, one often discretizes a continuous 2D environment into a regular grid with blocked and unblocked cells and then finds shortest paths for the agents on the resulting grid graph. Shortest grid paths, of course, are not necessarily true shortest paths in the continuous 2D environment. In this article, we therefore study how much longer a shortest grid path can be than a corresponding true shortest path on all regular grids with blocked and unblocked cells that tessellate continuous 2D environments. We study 5 different vertex connectivities that result from both different tessellations and different definitions of the neighbors of a vertex. Our path-length analysis yields either tight or asymptotically tight worst-case bounds in a unified framework. Our results show that the percentage by which a shortest grid path can be longer than a corresponding true shortest path decreases as the vertex connectivity increases. Our path-length analysis is topical because it determines the largest path-length reduction possible for any-angle path-planning algorithms (and thus their benefit), a class of path-planning algorithms in artificial intelligence and robotics that has become popular.

Posted in: Artificial Intelligence, Robot task planning , Tagged: