Transfering RL knowledge to other tasks by transferring components of RL

November 10, 2023 09:49 ,

Rex G. Liu, Michael J. Frank, Hierarchical clustering optimizes the tradeoff between compositionality and expressivity of task structures for flexible reinforcement learning, Artificial Intelligence, Volume 312, 2022 DOI: 10.1016/j.artint.2022.103770.

A hallmark of human intelligence, but challenging for reinforcement learning (RL) agents, is the ability to compositionally generalise, that is, to recompose familiar knowledge components in novel ways to solve new problems. For instance, when navigating in a city, one needs to know the location of the destination and how to operate a vehicle to get there, whether it be pedalling a bike or operating a car. In RL, these correspond to the reward function and transition function, respectively. To compositionally generalize, these two components need to be transferable independently of each other: multiple modes of transport can reach the same goal, and any given mode can be used to reach multiple destinations. Yet there are also instances where it can be helpful to learn and transfer entire structures, jointly representing goals and transitions, particularly whenever these recur in natural tasks (e.g., given a suggestion to get ice cream, one might prefer to bike, even in new towns). Prior theoretical work has explored how, in model-based RL, agents can learn and generalize task components (transition and reward functions). But a satisfactory account for how a single agent can simultaneously satisfy the two competing demands is still lacking. Here, we propose a hierarchical RL agent that learns and transfers individual task components as well as entire structures (particular compositions of components) by inferring both through a non-parametric Bayesian model of the task. It maintains a factorised representation of task components through a hierarchical Dirichlet process, but it also represents different possible covariances between these components through a standard Dirichlet process. We validate our approach on a variety of navigation tasks covering a wide range of statistical correlations between task components and show that it can also improve generalisation and transfer in more complex, hierarchical tasks with goal/subgoal structures. Finally, we end with a discussion of our work including how this clustering algorithm could conceivably be implemented by cortico-striatal gating circuits in the brain.

Posted in: Uncategorized , Tagged: , ,

A RRT-based method that addresses combined task and motion planning

November 10, 2023 09:38 ,

Riccardo Caccavale, Alberto Finzi, A rapidly-exploring random trees approach to combined task and motion planning, Robotics and Autonomous Systems, Volume 157, 2022 DOI: 10.1016/j.robot.2022.104238.

Task and motion planning in robotics are typically addressed by separated intertwined methods. Task planners generate abstract high-level actions to be executed, while motion planners provide the associated discrete movements in the configuration space satisfying kinodynamic constraints. However, these two planning processes are strictly dependent, therefore the problem of combining task and motion planning with a uniform approach is very relevant. In this work, we tackle this issue by proposing a RRT-based method that addresses combined task and motion planning. Our approach relies on a combined metric space where both symbolic (task) and sub-symbolic (motion) spaces are represented. The associated notion of distance is then exploited by a RRT-based planner to generate a plan that includes both symbolic actions and feasible movements in the configuration space. The proposed method is assessed in several case studies provided by a real-world hospital logistic scenario, where an omni-directional mobile robot is involved in navigation and transportation tasks.

Posted in: Robot motion planning, Robot task planning , Tagged:

Using results from belief-based planning for Bayesian inference in robotics

November 10, 2023 09:35 ,

Farhi, E.I., Indelman, V., Bayesian incremental inference update by re-using calculations from belief space planning: a new paradigm, Auton Robot 46, 783\u2013816 (2022). DOI: 10.1007/s10514-022-10045-w.

Inference and decision making under uncertainty are key processes in every autonomous system and numerous robotic problems. In recent years, the similarities between inference and decision making triggered much work, from developing unified computational frameworks to pondering about the duality between the two. In spite of these efforts, inference and control, as well as inference and belief space planning (BSP) are still treated as two separate processes. In this paper we propose a paradigm shift, a novel approach which deviates from conventional Bayesian inference and utilizes the similarities between inference and BSP. We make the key observation that inference can be efficiently updated using predictions made during the decision making stage, even in light of inconsistent data association between the two. We developed a two staged process that implements our novel approach and updates inference using calculations from the precursory planning phase. Using autonomous navigation in an unknown environment along with iSAM2 efficient methodologies as a test case, we benchmarked our novel approach against standard Bayesian inference, both with synthetic and real-world data (KITTI dataset). Results indicate that not only our approach improves running time by at least a factor of two while providing the same estimation accuracy, but it also alleviates the computational burden of state dimensionality and loop closures.

Posted in: Mobile Robot Localization, Mobile robot mapping , Tagged: ,

Real-time and Bayesian-enabled ICP for mobile robot localization and mapping in a Bayesian framework

November 10, 2023 09:31 ,

Maken FA, Ramos F, Ott L. , Bayesian iterative closest point for mobile robot localization, The International Journal of Robotics Research. 2022;41(9-10):851-874 DOI: 10.1177/02783649221101417.

Accurate localization of a robot in a known environment is a fundamental capability for successfully performing path planning, manipulation, and grasping tasks. Particle filters, also known as Monte Carlo localization (MCL), are a commonly used method to determine the robot\u2019s pose within its environment. For ground robots, noisy wheel odometry readings are typically used as a motion model to predict the vehicle\u2019s location. Such a motion model requires tuning of various parameters based on terrain and robot type. However, such an ego-motion estimation is not always available for all platforms. Scan matching using the iterative closest point (ICP) algorithm is a popular alternative approach, providing ego-motion estimates for localization. Iterative closest point computes a point estimate of the transformation between two poses given point clouds captured at these locations. Being a point estimate method, ICP does not deal with the uncertainties in the scan alignment process, which may arise due to sensor noise, partial overlap, or the existence of multiple solutions. Another challenge for ICP is the high computational cost required to align two large point clouds, limiting its applicability to less dynamic problems. In this paper, we address these challenges by leveraging recent advances in probabilistic inference. Specifically, we first address the run-time issue and propose SGD-ICP, which employs stochastic gradient descent (SGD) to solve the optimization problem of ICP. Next, we leverage SGD-ICP to obtain a distribution over transformations and propose a Markov Chain Monte Carlo method using stochastic gradient Langevin dynamics (SGLD) updates. Our ICP variant, termed Bayesian-ICP, is a full Bayesian solution to the problem. To demonstrate the benefits of Bayesian-ICP for mobile robotic applications, we propose an adaptive motion model employing Bayesian-ICP to produce proposal distributions for Monte Carlo Localization. Experiments using both Kinect and 3D LiDAR data show that our proposed SGD-ICP method achieves the same solution quality as standard ICP while being significantly more efficient. We then demonstrate empirically that Bayesian-ICP can produce accurate distributions over pose transformations and is fast enough for online applications. Finally, using Bayesian-ICP as a motion model alleviates the need to tune the motion model parameters from odometry, resulting in better-calibrated localization uncertainty.

Posted in: Mobile Robot Localization , Tagged: ,

Adaptation of industrial robots to variations in tasks through RL

November 10, 2023 09:27 ,

Tian Yu, Qing Chang, User-guided motion planning with reinforcement learning for human-robot collaboration in smart manufacturing, Expert Systems with Applications, Volume 209, 2022 DOI: 10.1016/j.eswa.2022.118291.

In today\u2019s manufacturing system, robots are expected to perform increasingly complex manipulation tasks in collaboration with humans. However, current industrial robots are still largely preprogrammed with very little autonomy and still required to be reprogramed by robotics experts for even slightly changed tasks. Therefore, it is highly desirable that robots can adapt to certain task changes with motion planning strategies to easily work with non-robotic experts in manufacturing environments. In this paper, we propose a user-guided motion planning algorithm in combination with reinforcement learning (RL) method to enable robots automatically generate their motion plans for new tasks by learning from a few kinesthetic human demonstrations. Features of common human demonstrated tasks in a specific application environment, e.g., desk assembly or warehouse loading/unloading are abstracted and saved in a library. The definition of semantical similarity between features in the library and features of a new task is proposed and further used to construct the reward function in RL. To achieve an adaptive motion plan facing task changes or new task requirements, features embedded in the library are mapped to appropriate task segments based on the trained motion planning policy using Q-learning. A new task can be either learned as a combination of a few features in the library or a requirement for further human demonstration if the current library is insufficient for the new task. We evaluate our approach on a 6 DOF UR5e robot on multiple tasks and scenarios and show the effectiveness of our method with respect to different scenarios.

Posted in: Applications of reinforcement learning to robots , Tagged: , ,

Reducing outliers in time series with singular spectrum analysis and use of deep learning for change detection

November 10, 2023 09:22 ,

Muktesh Gupta, Rajesh Wadhvani, Akhtar Rasool, Real-time Change-Point Detection: A deep neural network-based adaptive approach for detecting changes in multivariate time series data, Expert Systems with Applications, Volume 209, 2022 DOI: 10.1016/j.eswa.2022.118260.

The behavior of a time series may be affected by various factors. Changes in mean, variance, frequency, and auto-correlation are the most common. Change-Point Detection (CPD) aims to track down abrupt statistical characteristic changes in time series that can benefit many applications in different domains. As demonstrated in recently introduced CPD methodologies, deep learning approaches have the potential to identify more subtle changes. However, due to improper handling of data and insufficient training, these methodologies generate more false alarms and are not efficient enough in detecting change-points. In real-time CPD algorithms, preprocessed data plays a vital role in increasing the algorithm\u2019s efficiency and minimizing false alarm rates. Therefore, preprocessing of data should be a part of the algorithm, but in the existing methods, preprocessing of data is done initially, and then the whole dataset is passed to the CPD algorithm. A new three-phase architecture is proposed to address this issue, in which all phases, from preprocessing to CPD, work in an adaptive manner. The phases are integrated into a pipeline, allowing the algorithm to work in real-time. Our proposed strategy performs optimally and consistently based on performance metrics resulting from experiments on real-world datasets and artifacts. This work effectively addresses the issue of non-stationary data normalization using deep learning approaches. To reduce noise and outliers from the data, a recursive version of singular spectrum analysis is introduced. It is demonstrated that the method\u2019s performance has significantly improved by combining adaptive preprocessing with deep learning CPD techniques.

NOTE: See also C. Ma, L. Zhang, W. Pedrycz and W. Lu, “The Long-Term Prediction of Time Series: A Granular Computing-Based Design Approach,” in IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 52, no. 10, pp. 6326-6338, Oct. 2022, doi: 10.1109/TSMC.2022.3144395.

See also https://babel.isa.uma.es/kipr/?p=1548

Posted in: Probability and statistics , Tagged: , ,

Non-parameterical detection of regimes in time series data (SODA), and its use in fuzzy forecasting

November 10, 2023 09:17 ,

Shivani Pant, Sanjay Kumar, IFS and SODA based computational method for fuzzy time series forecasting, Expert Systems with Applications, Volume 209, 2022 DOI: 10.1016/j.eswa.2022.118213.

Time series forecasting has fascinated a great deal of interest from various research communities due to its wide applications in medicine, economics, finance, engineering and many other crucial fields. Various studies in past have shown that intuitionistic fuzzy sets (IFSs) not only handle non-stochastic non-determinism in time series forecasting but also enhance accuracy in forecasted outputs. Clustering is another one of the methods that improves accuracy of time series forecasting. The contribution of this research work is a novel computational fuzzy time series (FTS) forecasting method which relies on IFSs and self-organized direction aware (SODA) approach of clustering. The usage of SODA aids in making the proposed FTS forecasting method as autonomous as feasible, as it does not require human intervention or prior knowledge of the data. Forecasted outputs in proposed FTS forecasting method are computed using a weighted formula and weights are optimized using grey wolf optimization (GWO) method. Proposed FTS is applied to forecast enrolments of the University of Alabama and market price of State Bank of India (SBI) share at Bombay stock exchange (BSE), India and performance is compared in terms of root mean square error (RMSE), average forecasting error (AFE) and mean absolute deviation (MAD). Goodness of the proposed FTS forecasting method in forecasting enrolments of the University of Alabama and market price of SBI share is also tested using coefficient of correlation and determination, criteria of Akaike and Bayesian information.

See also https://babel.isa.uma.es/kipr/?p=1550

Posted in: Probability and statistics , Tagged: ,

On the extended use of RL for navigation in UAVs

November 3, 2023 10:07 ,

Fadi AlMahamid, Katarina Grolinger, Autonomous Unmanned Aerial Vehicle navigation using Reinforcement Learning: A systematic review, Engineering Applications of Artificial Intelligence, Volume 115, 2022 DOI: 10.1016/j.engappai.2022.105321.

There is an increasing demand for using Unmanned Aerial Vehicle (UAV), known as drones, in different applications such as packages delivery, traffic monitoring, search and rescue operations, and military combat engagements. In all of these applications, the UAV is used to navigate the environment autonomously \u2014 without human interaction, perform specific tasks and avoid obstacles. Autonomous UAV navigation is commonly accomplished using Reinforcement Learning (RL), where agents act as experts in a domain to navigate the environment while avoiding obstacles. Understanding the navigation environment and algorithmic limitations plays an essential role in choosing the appropriate RL algorithm to solve the navigation problem effectively. Consequently, this study first identifies the main UAV navigation tasks and discusses navigation frameworks and simulation software. Next, RL algorithms are classified and discussed based on the environment, algorithm characteristics, abilities, and applications in different UAV navigation problems, which will help the practitioners and researchers select the appropriate RL algorithms for their UAV navigation use cases. Moreover, identified gaps and opportunities will drive UAV navigation research.

Posted in: Applications of reinforcement learning to robots , Tagged: , , ,

Current state of the practical use of real-time systems, got through industry questionnaires

November 3, 2023 09:54 ,

Akesson, B., Nasri, M., Nelissen, G. et al. A comprehensive survey of industry practice in real-time systems, Real-Time Syst 58, 358\u2013398 (2022) DOI: 10.1007/s11241-021-09376-1.

This paper presents results and observations from a survey of 120 industry practitioners in the field of real-time embedded systems. The survey provides insights into the characteristics of the systems being developed today and identifies important trends for the future. It extends the results from the survey data to the broader population that it is representative of, and discusses significant differences between application domains. The survey aims to inform both academics and practitioners, helping to avoid divergence between industry practice and academic research. The value of this research is highlighted by a study showing that the aggregate findings of the survey are not common knowledge in the real-time systems community.

Posted in: Real-Time Systems , Tagged: ,

A remote Matlab laboratory for LTI system identification

November 3, 2023 09:32 ,

Z. Lei, H. Zhou, W. Hu and G. -P. Liu, Teaching and Comprehensive Learning With Remote Laboratories and MATLAB for an Undergraduate System Identification Course, EEE Transactions on Education, vol. 65, no. 3, pp. 402-408, Aug. 2022 DOI: 10.1109/TE.2021.3123302.

Contribution: This article introduces the teaching and learning with remote laboratories and MATLAB for an undergraduate system identification (SI) course, which can be employed for students at the advanced level with a control background. Background: SI has been widely used in all engineering fields; thus, the SI course that includes complex theories, concepts, and formulas is crucial for engineering education. Constraints, such as time, space, cost, and maintenance work, pose limitations for conventional laboratories, and current remote laboratories may not offer experiences to enhance control-oriented practical skills. Intended Outcomes: The proposed teaching and learning using remote laboratories is intended to facilitate the understanding of theories and concepts, and enhance the ability of design and implementation of control algorithms, the conducting of experiments, data collecting, data analysis, and the conducting of SI with MATLAB. Application Design: In the classroom teaching, theoretical lectures regarding SI are delivered to students by the teacher, along with the classroom demonstration with the networked control system laboratory for online experimentation. Then, the laboratory work is required to be completed by the students using the remote laboratory. A tailored laboratory report is supposed to be handed in by each student after the experimentation. Findings: The effectiveness of the proposed method was evaluated through the analysis of student performance and student responses to surveys. The student performance analysis indicates that the application of the remote laboratories is effective, and the feedback from students shows that they can benefit from the application of remote laboratories, and they would like the remote laboratories to be expanded to other courses.

Posted in: Control Engineering , Tagged: , , ,