Adapting a shared teleoperation system to network delays

September 4, 2025 08:42 ,

B. Güleçyüz et al., Enhancing Shared Autonomy in Teleoperation Under Network Delay: Transparency- and Confidence-Aware Arbitration, IEEE Robotics and Automation Letters, vol. 10, no. 10, pp. 9654-9661, Oct. 2025, 10.1109/LRA.2025.3596436.

Shared autonomy bridges human expertise with machine intelligence, yet existing approaches often overlook the impact of teleoperation delays. To address this gap, we propose a novel shared autonomy approach that enables robots to gradually learn from teleoperated demonstrations while adapting to network delays. Our method improves intent prediction by accounting for delayed feedback to the human operator and adjusts the arbitration function to balance reduced human confidence due to delay with confidence in learned autonomy. To ensure system stability, which might be compromised by delay and arbitration of human and autonomy control forces, we introduce a three-port extension of the Time-Domain Passivity Approach with Energy Reflection (TDPA-ER). Experimental validation with 12 participants demonstrated improvements in intent prediction accuracy, task performance, and the quality of final learned autonomy, highlighting the potential of our approach to enhance teleoperation and learning quality in remote environments.

Posted in: Human teleoperation , Tagged:

A review of cognitive costs of decision making

September 4, 2025 08:36 ,

Christin Schulze, Ada Aka, Daniel M. Bartels, Stefan F. Bucher, Jake R. Embrey, Todd M. Gureckis, Gerald Häubl, Mark K. Ho, Ian Krajbich, Alexander K. Moore, Gabriele Oettingen, Joan D.K. Ongchoco, Ryan Oprea, Nicholas Reinholtz, Ben R. Newell, A timeline of cognitive costs in decision-making, Trends in Cognitive Sciences, Volume 29, Issue 9, 2025, Pages 827-839, 10.1016/j.tics.2025.04.004.

Recent research from economics, psychology, cognitive science, computer science, and marketing is increasingly interested in the idea that people face cognitive costs when making decisions. Reviewing and synthesizing this research, we develop a framework of cognitive costs that organizes concepts along a temporal dimension and maps out when costs occur in the decision-making process and how they impact decisions. Our unifying framework broadens the scope of research on cognitive costs to a wider timeline of cognitive processing. We identify implications and recommendations emerging from our framework for intervening on behavior to tackle some of the most pressing issues of our day, from improving health and saving decisions to mitigating the consequences of climate change.

Posted in: Psycho-physiological bases of engineering , Tagged:

Social learning is compatible with reward-based decision making

September 4, 2025 08:33 ,

David Schultner, Lucas Molleman, Björn Lindström, Reward is enough for social learning, Trends in Cognitive Sciences, Volume 29, Issue 9, 2025, Pages 787-789, 10.1016/j.tics.2025.06.012.

Adaptive behaviour relies on selective social learning, yet the mechanisms underlying this capacity remain debated. A new account demonstrates that key strategies can emerge through reward-based learning of social features, explaining the widely observed flexibility of social learning and illuminating the cognitive basis of cultural evolution.

Posted in: Psycho-physiological bases of engineering , Tagged:

On the theoretical convergence of Q-learning when the environment is not stationary

September 4, 2025 08:31 ,

Diogo S. Carvalho, Pedro A. Santos, Francisco S. Melo, Reinforcement learning in convergently non-stationary environments: Feudal hierarchies and learned representations, Artificial Intelligence, Volume 347, 2025, 10.1016/j.artint.2025.104382.

We study the convergence of Q-learning-based methods in convergently non-stationary environments, particularly in the context of hierarchical reinforcement learning and of dynamic features encountered in deep reinforcement learning. We demonstrate that Q-learning achieves convergence in tabular representations when applied to convergently non-stationary dynamics, such as the ones arising in a feudal hierarchical setting. Additionally, we establish convergence for Q-learning-based deep reinforcement learning methods with convergently non-stationary features, such as the ones arising in representation-based settings. Our findings offer theoretical support for the application of Q-learning in these complex scenarios and present methodologies for extending established theoretical results from standard cases to their convergently non-stationary counterparts.

Posted in: Reinforcement learning theory , Tagged: ,

Improving the generalization of robotic RL by inspiration in the humman motion control system

September 4, 2025 08:28 ,

P. Zhang, Z. Hua and J. Ding, A Central Motor System Inspired Pretraining Reinforcement Learning for Robotic Control, IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 55, no. 9, pp. 6285-6298, Sept. 2025, 10.1109/TSMC.2025.3577698.

Robots typically encounter diverse tasks, bringing a significant challenge for motion control. Pretraining reinforcement learning (PRL) enables robots to adapt quickly to various tasks by exploiting reusable skills. The existing PRL methods often rely on datasets and human expert knowledge, struggle to discover diverse and dynamic skills, and exhibit generalization and adaptability to different types of robots and downstream tasks. This article proposes a novel PRL algorithm based on the central motor system mechanisms, which can discover diverse and dynamic skills without relying on data and expert knowledge, effectively enabling robots to tackle different types of downstream tasks. Inspired by the cerebellum’s role in balance control and skill storage within the central motor system, an intrinsic fused reward is introduced to explore dynamic skills and eliminate dependence on data and expert knowledge during pretraining. Drawing from the basal ganglia’s function in motor programming, a discrete skill encoding method is designed to increase the diversity of discovered skills, improving the performance of complex robots in challenging environments. Furthermore, incorporating the basal ganglia’s role in motor regulation, a skill activity function is proposed to generate skills at varying dynamic levels, thereby improving the adaptability of robots in multiple downstream tasks. The effectiveness of the proposed algorithm has been demonstrated through simulation experiments on four different morphological robots across multiple downstream tasks.

Posted in: Applications of reinforcement learning to robots, Psycho-physiological bases of engineering , Tagged: ,

Stacking multiple MDPs in an abstraction hierarchy to better solve RL

September 4, 2025 08:21 ,

Roberto Cipollone, Marco Favorito, Flavio Maiorana, Giuseppe De Giacomo, Luca Iocchi, Fabio Patrizi, Exploiting robot abstractions in episodic RL via reward shaping and heuristics, Robotics and Autonomous Systems, Volume 193, 2025, 10.1016/j.robot.2025.105116.

One major limitation to the applicability of Reinforcement Learning (RL) to many domains of practical relevance, in particular in robotic applications, is the large number of samples required to learn an optimal policy. To address this problem and improve learning efficiency, we consider a linear hierarchy of abstraction layers of the Markov Decision Process (MDP) underlying the target domain. Each layer is an MDP representing a coarser model of the one immediately below in the hierarchy. In this work, we propose novel techniques to automatically define Reward Shaping and Reward Heuristic functions that are based on the solution obtained at a higher level of abstraction and provide rewards to the finer (possibly the concrete) MDP at the lower level, thus inducing an exploration heuristic that can effectively guide the learning process in the more complex domain. In contrast with other works in Hierarchical RL, our technique imposes fewer requirements on the design of the abstract models and is tolerant to modeling errors, thus making the proposed approach practical. We formally analyze the relationship between the abstract models and the exploration heuristic induced in the lower-level domain, we prove that the method guarantees optimal convergence, and finally demonstrate its effectiveness experimentally in several complex robotic domains.

Posted in: Applications of reinforcement learning to robots , Tagged: ,

How biology uses primary rewards coming from basic physiological signals and proxy rewards, more immediate (predicting primary rewards) as shaping rewards

September 3, 2025 16:18 ,

Lilian A. Weber, Debbie M. Yee, Dana M. Small, and Frederike H. Petzschner, The interoceptive origin of reinforcement learning, IEEE Robotics and Automation Letters, vol. 10, no. 8, pp. 7723-7730, Aug. 2025, 10.1016/j.tics.2025.05.008.

Rewards play a crucial role in sculpting all motivated behavior. Traditionally, research on reinforcement learning has centered on how rewards guide learning and decision-making. Here, we examine the origins of rewards themselves. Specifically, we discuss that the critical signal sustaining reinforcement for food is generated internally and subliminally during the process of digestion. As such, a shift in our understanding of primary rewards as an immediate sensory gratification to a state-dependent evaluation of an action’s impact on vital phys- iological processes is called for. We integrate this perspective into a revised reinforcement learning framework that recognizes the subliminal nature of bio-logical rewards and their dependency on internal states and goals.

Posted in: Psycho-physiological bases of engineering , Tagged: ,

Including “fear” and “curiosity” in RL applied to robot navigation

July 4, 2025 09:07 ,

D. Hu, L. Mo, J. Wu and C. Huang, “Feariosity”-Guided Reinforcement Learning for Safe and Efficient Autonomous End-to-End Navigation, IEEE Robotics and Automation Letters, vol. 10, no. 8, pp. 7723-7730, Aug. 2025, 10.1109/LRA.2025.3577523.

End-to-end navigation strategies using reinforcement learning (RL) can improve the adaptability and autonomy of Autonomous ground vehicles (AGVs) in complex environments. However, RL still faces challenges in data efficiency and safety. Neuroscientific and psychological research shows that during exploration, the brain balances between fear and curiosity, a critical process for survival and adaptation in dangerous environments. Inspired by this scientific insight, we propose the “Feariosity” model, which integrates fear and curiosity model to simulate the complex psychological dynamics organisms experience during exploration. Based on this model, we developed an innovative policy constraint method that evaluates potential hazards and applies necessary safety constraints while encouraging exploration of unknown areas. Additionally, we designed a new experience replay mechanism that quantifies the threat and unknown level of data, optimizing their usage probability. Extensive experiments in both simulation and real-world scenarios demonstrate that the proposed method significantly improves data efficiency, asymptotic performance during training. Furthermore, it achieves higher success rates, driving efficiency, and robustness in deployment. This also highlights the key role of mimicking biological neural and psychological mechanisms in improving the safety and efficiency through RL.

Posted in: Applications of reinforcement learning to control engineering, Psycho-physiological bases of engineering , Tagged: ,

It seems that human predictive brain works by predicting at the abstract level, not at the sensory level

July 4, 2025 09:02 ,

Kaitlyn M. Gabhart, Yihan (Sophy) Xiong, André M. Bastos, Predictive coding: a more cognitive process than we thought?, Trends in Cognitive Sciences, Volume 29, Issue 7, 2025, Pages 627-640, 10.1016/j.tics.2025.01.012.

In predictive coding (PC), higher-order brain areas generate predictions that are sent to lower-order sensory areas. Top-down predictions are compared with bottom-up sensory data, and mismatches evoke prediction errors. In PC, the prediction errors are encoded in layer 2/3 pyramidal neurons of sensory cortex that feed forward. The PC model has been tested with multiple recording modalities using the global–local oddball paradigm. Consistent with PC, neuroimaging studies reported prediction error responses in sensory and higher-order areas. However, recent studies of neuronal spiking suggest that genuine prediction errors emerge in prefrontal cortex (PFC). This implies that predictive processing is a more cognitive than sensory-based mechanism – an observation that challenges PC and better aligns with a framework we call predictive routing (PR).

Posted in: Psycho-physiological bases of engineering , Tagged:

Evidences of the dimensionality reduction and augmentation performed by the brain

July 4, 2025 08:58 ,

Casper Kerrén, Daniel Reznik, Christian F. Doeller, Benjamin J. Griffiths, Exploring the role of dimensionality transformation in episodic memory, Trends in Cognitive Sciences, Volume 29, Issue 7, 2025, Pages 614-626, 10.1016/j.tics.2025.01.007.

Episodic memory must accomplish two adversarial goals: encoding and storing a multitude of experiences without exceeding the finite neuronal structure of the brain, and recalling memories in vivid detail. Dimensionality reduction and expansion (‘dimensionality transformation’) enable the brain to meet these demands. Reduction compresses sensory input into simplified, storable codes, while expansion reconstructs vivid details. Although these processes are essential to memory, their neural mechanisms for episodic memory remain unclear. Drawing on recent insights from cognitive psychology, systems neuroscience, and neuroanatomy, we propose two accounts of how dimensionality transformation occurs in the brain: structurally (via corticohippocampal pathways) and functionally (through neural oscillations). By examining cross-species evidence, we highlight neural mechanisms that may support episodic memory and identify crucial questions for future research.

Posted in: Psycho-physiological bases of engineering , Tagged: