Author Archives: Juan-antonio Fernández-madrigal

Interesting survey of existing sim-to-real gap in RL in the context of humanoid robots

March 20, 2025 15:41 ,

D. Kim, H. Lee, J. Cha and J. Park, Bridging the Reality Gap: Analyzing Sim-to-Real Transfer Techniques for Reinforcement Learning in Humanoid Bipedal Locomotion, IEEE Robotics & Automation Magazine, vol. 32, no. 1, pp. 49-58, March 2025 10.1109/MRA.2024.3505784.

Reinforcement learning (RL) offers a promising solution for controlling humanoid robots, particularly for bipedal locomotion, by learning adaptive and flexible control strategies. However, direct RL application is hindered by time-consuming trial-and-error processes, necessitating training in simulation before real-world transfer. This introduces a reality gap that degrades performance. Although various methods have been proposed for sim-to-real transfer, they have not been validated on a consistent hardware platform, making it difficult to determine which components are key to overcoming the reality gap. In contrast, we systematically evaluate techniques to enhance RL policy robustness during sim-to-real transfer by controlling variables and comparing them on a single robot to isolate and analyze the impact of each technique. These techniques include dynamics randomization, state history usage, noise/bias/delay modeling, state selection, perturbations, and network size. We quantitatively assess the reality gap by simulating diverse conditions and conducting experiments on real hardware. Our findings provide insights into bridging the reality gap, advancing robust RL-trained humanoid robots for real-world applications.

Posted in: Applications of reinforcement learning to robots , Tagged:

A new perspective of considering convex optimization problems based on electric circuit theory

March 6, 2025 10:41 ,

Stephen P. Boyd, Tetiana Parshakova, Ernest K. Ryu, Jaewook J. Suh, Optimization Algorithm Design via Electric Circuits, NeurIPS 2024 spotlight, 25 Sept 2024, Last Modified: 14 Jan 2025, https://openreview.net/forum?id=9Jmt1eER9P.

We present a novel methodology for convex optimization algorithm design using ideas from electric RLC circuits. Given an optimization problem, the first stage of the methodology is to design an appropriate electric circuit whose continuous-time dynamics converge to the solution of the optimization problem at hand. Then, the second stage is an automated, computer-assisted discretization of the continuous-time dynamics, yielding a provably convergent discrete-time algorithm. Our methodology recovers many classical (distributed) optimization algorithms and enables users to quickly design and explore a wide range of new algorithms with convergence guarantees.

Posted in: Mathematics , Tagged: , ,

On the innate ability of vertebrates for number recognition and the one of distinguishing ratios of numbers

March 6, 2025 10:33 ,

Elena Lorenzi, Dmitry Kobylkov, Giorgio Vallortigara, Is there an innate sense of number in the brain?, Cerebral Cortex, Volume 35, Issue 2, February 2025, DOI: 10.1093/cercor/bhaf004.

The approximate number system or «sense of number» is a crucial, presymbolic mechanism enabling animals to estimate quantities, which is essential for survival in various contexts (eg estimating numerosities of social companions, prey, predators, and so on). Behavioral studies indicate that a sense of number is widespread across vertebrates and invertebrates. Specific brain regions such as the intraparietal sulcus and prefrontal cortex in primates, or equivalent areas in birds and fish, are involved in numerical estimation, and their activity is modulated by the ratio of quantities. Data gathered across species strongly suggest similar evolutionary pressures for number estimation pointing to a likely common origin, at least across vertebrates. On the other hand, few studies have investigated the origins of the sense of number. Recent findings, however, have shown that numerosity-selective neurons exist in newborn animals, such as domestic chicks and zebrafish, supporting the hypothesis of an innateness of the approximate number system. Control-rearing experiments on visually naïve animals further support the notion that the sense of number is innate and does not need any specific instructive experience in order to be triggered.

Posted in: Psycho-physiological bases of engineering , Tagged:

It seems that the human brain working memory uses pointers

March 6, 2025 10:21 ,

Edward Awh, Edward K. Vogel, Working memory needs pointers, Trends in Cognitive Sciences, Volume 29, Issue 3, 2025, Pages 230-241, DOI: 10.1016/j.tics.2024.12.006.

Cognitive neuroscience has converged on a definition of working memory (WM) as a capacity-limited system that maintains highly accessible representations via stimulus-specific neural patterns. We argue that this standard definition may be incomplete. We highlight the fundamental need to recognize specific instances or tokens and to bind those tokens to the surrounding context. We propose that contextual binding is supported by spatiotemporal ‘pointers’ and that pointers are the source of neural signals that track the number of stored items, independent of their content. These content-independent pointers may provide a productive perspective for understanding item-based capacity limits in WM and the role of WM as a gateway for long-term storage.

Posted in: Psycho-physiological bases of engineering , Tagged:

Planning tasks under uncertainty that have a maximum time to be finished

March 6, 2025 10:17 ,

Michal Staniaszek, Lara Brudermüller, Yang You, Raunak Bhattacharyya, Bruno Lacerda, Nick Hawes, Time-bounded planning with uncertain task duration distributions, Robotics and Autonomous Systems, Volume 186, 2025, DOI: 10.1016/j.robot.2025.104926.

We consider planning problems where a robot must gather reward by completing tasks at each of a large set of locations while constrained by a time bound. Our focus is problems where the context under which each task will be executed can be predicted, but is not known in advance. Here, the term context refers to the conditions under which the task is executed, and can be related to the robot’s internal state (e.g., how well it is localised?), or the environment itself (e.g., how dirty is the floor the robot must clean?). This context has an impact on the time required to execute the task, which we model probabilistically. We model the problem of time-bounded planning for tasks executed under uncertain contexts as a Markov decision process with discrete time in the state, and propose variants on this model which allow adaptation to different robotics domains. Due to the intractability of the general model, we propose simplifications to allow planning in large domains. The key idea behind these simplifications is constraining navigation using a solution to the travelling salesperson problem. We evaluate our models on maps generated from real-world environments and consider two domains with different characteristics: UV disinfection, and cleaning. We evaluate the effect of model variants and simplifications on performance, and show that policies obtained for our models outperform a rule-based baseline, as well as a model which does not consider context. We also evaluate our models in a real robot experiment where a quadruped performs simulated inspection tasks in an industrial environment.

Posted in: Robot task planning , Tagged:

RL for multiple tasks in the case of quadrotors and a short state of the art about the general problem

February 20, 2025 10:24 ,

J. Xing, I. Geles, Y. Song, E. Aljalbout and D. Scaramuzza, Multi-Task Reinforcement Learning for Quadrotors, IEEE Robotics and Automation Letters, vol. 10, no. 3, pp. 2112-2119, March 2025, DOI: 10.1109/LRA.2024.3520894.

Reinforcement learning (RL) has shown great effectiveness in quadrotor control, enabling specialized policies to develop even human-champion-level performance in single-task scenarios. However, these specialized policies often struggle with novel tasks, requiring a complete retraining of the policy from scratch. To address this limitation, this paper presents a novel multi-task reinforcement learning (MTRL) framework tailored for quadrotor control, leveraging the shared physical dynamics of the platform to enhance sample efficiency and task performance. By employing a multi-critic architecture and shared task encoders, our framework facilitates knowledge transfer across tasks, enabling a single policy to execute diverse maneuvers, including high-speed stabilization, velocity tracking, and autonomous racing. Our experimental results, validated both in simulation and real-world scenarios, demonstrate that our framework outperforms baseline approaches in terms of sample efficiency and overall task performance. Video is available at https://youtu.be/HfK9UT1OVnY.

Posted in: Applications of reinforcement learning to control engineering, Applications of reinforcement learning to robots , Tagged:

An adaptive KF for estimating angles from IMUs

February 13, 2025 11:51 ,

Zolfa Anvari, Ali Mirhaghgoo, Yasin Salehi, Real-time angle estimation in IMU sensors: An adaptive Kalman filter approach with forgetting factor, Mechatronics, Volume 106, 2025, DOI: 10.1016/j.mechatronics.2024.103280.

In recent years, the applications of Inertial Measurement Unit (IMU) sensors have witnessed significant growth across multiple fields. However, challenges regarding angle estimation using these sensors have emerged, primarily because of the lack of accuracy in accelerometer-based dynamic motion measurements and the associated bias and error accumulation when combined with gyroscope integration. Consequently, the Kalman filter has become a popular choice for addressing these issues, as it enables the sensor to operate dynamically. Despite its widespread use, the Kalman filter requires precise noise statistics estimation for optimal noise cancellation. To accommodate this requirement, adaptive Kalman filter algorithms have been developed for estimating zero-mean Gaussian process matrix (Q) and measurement matrix (R) variances. This study introduces a real-time adaptive approach that employs a forgetting factor to precisely estimate roll and pitch angles in a 6-axis IMU. The study’s novelty lies in its algorithm, which computes the forgetting factor based on the estimation error of the last samples in the sequence. Experimental results for roll angle indicate that, in response to a step change signal, this method achieves a 54%, 39%, and 70% reduction in RMS error relative to the raw sensor data, traditional Kalman filter, and a hybrid adaptive method, respectively. Moreover, this technique exhibits significant improvements in both fixed and sinusoidal conditions for roll and pitch angles, successfully carrying out tasks within required timescales without failures related to computation time.

Posted in: Bayesian filtering , Tagged: ,

On the reasons of the pervasiveness of the myth of meritocracy

February 6, 2025 13:15 ,

Ian R. Hadden, Céline Darnon, Lewis Doyle, Matthew J. Easterbrook, Sébastien Goudeau, Andrei Cimpian, Why the belief in meritocracy is so pervasive, Trends in Cognitive Sciences, Volume 29, Issue 2, 2025, Pages 101-104, DOI: 10.1016/j.tics.2024.12.008.

People worldwide tend to believe that their societies are more meritocratic than they actually are. We propose the belief in meritocracy is widespread because it is rooted in simple, seemingly obvious causal–explanatory intuitions. Our proposal suggests solutions for debunking the myth of meritocracy and increasing support for equity-oriented policies.

Posted in: Cognitive sciences , Tagged:

On the two-ways of learning language in humans: both abstracting detailed knowledge and refining still-only-abstract one

January 9, 2025 16:38 ,

Susan Goldin-Meadow, Inbal Arnon, Whole-to-part development in language creation, Trends in Cognitive Sciences, Volume 29, Issue 1, 2025, Pages 12-14, DOI: 10.1016/j.tics.2024.09.015.

Children approach language by learning parts and constructing wholes. But they can also first learn wholes and then discover parts. We demonstrate this understudied yet impactful process in children creating language without input. Whole-to-part learning thus need not be driven by hard-to-segment input and is a bias that children bring to language.

Posted in: Psycho-physiological bases of engineering , Tagged:

On the use of GPUs for parallelization of MPCs through the parallelization of symbolic mathematical expressions

January 9, 2025 16:29 ,

S. H. Jeon, S. Hong, H. J. Lee, C. Khazoom and S. Kim, CusADi: A GPU Parallelization Framework for Symbolic Expressions and Optimal Control, IEEE Robotics and Automation Letters, vol. 10, no. 2, pp. 899-906, Feb. 2025, DOI: 10.1109/LRA.2024.3512254.

The parallelism afforded by GPUs presents significant advantages in training controllers through reinforcement learning (RL). However, integrating model-based optimization into this process remains challenging due to the complexity of formulating and solving optimization problems across thousands of instances. In this work, we present CusADi , an extension of the casadi symbolic framework to support the parallelization of arbitrary closed-form expressions on GPUs with CUDA . We also formulate a closed-form approximation for solving general optimal control problems, enabling large-scale parallelization and evaluation of MPC controllers. Our results show a ten-fold speedup relative to similar MPC implementation on the CPU, and we demonstrate the use of CusADi for various applications, including parallel simulation, parameter sweeps, and policy training.

Posted in: Computer science , Tagged: , ,