Author Archives: Juan-antonio Fernández-madrigal

Current state of the practical use of real-time systems, got through industry questionnaires

November 3, 2023 09:54 ,

Akesson, B., Nasri, M., Nelissen, G. et al. A comprehensive survey of industry practice in real-time systems, Real-Time Syst 58, 358\u2013398 (2022) DOI: 10.1007/s11241-021-09376-1.

This paper presents results and observations from a survey of 120 industry practitioners in the field of real-time embedded systems. The survey provides insights into the characteristics of the systems being developed today and identifies important trends for the future. It extends the results from the survey data to the broader population that it is representative of, and discusses significant differences between application domains. The survey aims to inform both academics and practitioners, helping to avoid divergence between industry practice and academic research. The value of this research is highlighted by a study showing that the aggregate findings of the survey are not common knowledge in the real-time systems community.

Posted in: Real-Time Systems , Tagged: ,

A remote Matlab laboratory for LTI system identification

November 3, 2023 09:32 ,

Z. Lei, H. Zhou, W. Hu and G. -P. Liu, Teaching and Comprehensive Learning With Remote Laboratories and MATLAB for an Undergraduate System Identification Course, EEE Transactions on Education, vol. 65, no. 3, pp. 402-408, Aug. 2022 DOI: 10.1109/TE.2021.3123302.

Contribution: This article introduces the teaching and learning with remote laboratories and MATLAB for an undergraduate system identification (SI) course, which can be employed for students at the advanced level with a control background. Background: SI has been widely used in all engineering fields; thus, the SI course that includes complex theories, concepts, and formulas is crucial for engineering education. Constraints, such as time, space, cost, and maintenance work, pose limitations for conventional laboratories, and current remote laboratories may not offer experiences to enhance control-oriented practical skills. Intended Outcomes: The proposed teaching and learning using remote laboratories is intended to facilitate the understanding of theories and concepts, and enhance the ability of design and implementation of control algorithms, the conducting of experiments, data collecting, data analysis, and the conducting of SI with MATLAB. Application Design: In the classroom teaching, theoretical lectures regarding SI are delivered to students by the teacher, along with the classroom demonstration with the networked control system laboratory for online experimentation. Then, the laboratory work is required to be completed by the students using the remote laboratory. A tailored laboratory report is supposed to be handed in by each student after the experimentation. Findings: The effectiveness of the proposed method was evaluated through the analysis of student performance and student responses to surveys. The student performance analysis indicates that the application of the remote laboratories is effective, and the feedback from students shows that they can benefit from the application of remote laboratories, and they would like the remote laboratories to be expanded to other courses.

Posted in: Control Engineering , Tagged: , , ,

On the importance of the static structures + execution flow in learning programming languages

November 3, 2023 09:27 ,

B. Bettin, M. Jarvie-Eggart, K. S. Steelman and C. Wallace, Preparing First-Year Engineering Students to Think About Code: A Guided Inquiry Approach, IEEE Transactions on Education, vol. 65, no. 3, pp. 309-319, Aug. 2022 DOI: 10.1109/TE.2021.3140051.

In the wake of the so-called fourth industrial revolution, computer programming has become a foundational competency across engineering disciplines. Yet engineering students often resist the notion that computer programming is a skill relevant to their future profession. Here are presented two activities aimed at supporting the early development of engineering students\u2019 attitudes and abilities regarding programming in a first-year engineering course. Both activities offer students insights into the way programs are constructed, which have been identified as a source of confusion that may negatively affect acceptance. In the first activity, a structured, language-independent way to approach programming problems through guided questions was introduced, which has previously been used successfully in introductory computer science courses. The team hypothesized that guiding students through a structured reflection on how they construct programs for their class assignments might help reveal an understandable structure to them. Results showed that students in the intervention group scored nearly a full letter grade higher on the unit\u2019s final programming assessment than those in the control condition. The second activity aimed to help students recognize how their experience with MATLAB might help them interpret code in other programming languages. In the intervention group, students were asked to review and provide comments for code written in a variety of programming languages. A qualitative analysis of their reflections examined what skills students reported they used and, specifically, how prior MATLAB experience may have aided their ability to read and comment on the unfamiliar code. Overall, the ability to understand and recognize syntactic constructs was an essential skill in making sense of code written in unfamiliar programming languages. Syntactic constructs, lexical elements, and patterns were all recognized as essential landmarks used by students interpreting code they did not write, especially in new languages. Developing an understanding of the static structure and dynamic flow required of programs was also an essential skill which helped the students. Together, the results from the first activity and the insights gained from the second activity suggest that guided questions to build skills in reading code may help mitigate confusion about program construction, thereby better preparing engineering students for computing-intensive careers.

Posted in: Computer science , Tagged: ,

Doing a more intelligent exploration in RL based on measuring uncertainty through prediction

November 3, 2023 09:12 ,

Xiaoshu Zhou, Fei Zhu, Peiyao Zhao, Within the scope of prediction: Shaping intrinsic rewards via evaluating uncertainty, Expert Systems with Applications, Volume 206, 2022 DOI: 10.1016/j.eswa.2022.117775.

The agent of reinforcement learning based approaches needs to explore to learn more about the environment to seek optimal policy. However, simply increasing the frequency of stochastic exploration sometimes fails to work or even causes the agent to fall into traps. To solve the problem, it is essential to improve the quality of exploration. An approach, referred to as the scope of prediction based on uncertainty exploration (SPE), is proposed, taking advantage of the uncertainty mechanism and considering the stochasticity of prospecting. As by uncertainty mechanism, the unexpected states make more curiosity, the model derives higher uncertainty by projecting future scenarios to compare with the actual future to explore the world. The SPE method utilizes a prediction network to predict subsequent observations and calculates the mean squared difference value of the real observations and the following observations to measure uncertainty, encouraging the agent to explore unknown regions more effectively. Moreover, to reduce the noise interference caused by uncertainty, a reward-penalty model is developed to discriminate the noise by current observations and action prediction for future rewards to improve the interference ability against noise so that the agent can escape from the noisy region. Experiment results showed that deep reinforcement learning approaches equipped with SPE demonstrated significant improvements in simulated environments.

Posted in: Reinforcement learning theory , Tagged: , ,

Normal blindness to visible objects seems to be the result of limited-capacity prediction mechanisms in the brain

November 3, 2023 09:05 ,

Jeremy M. Wolfe, Anna Kosovicheva, Benjamin Wolfe, Normal blindness: when we Look But Fail To See, Trends in Cognitive Sciences, Volume 26, Issue 9, 2022, Pages 809-819 DOI: 10.1016/j.tics.2022.06.006.

Humans routinely miss important information that is \u2018right in front of our eyes\u2019, from overlooking typos in a paper to failing to see a cyclist in an intersection. Recent studies on these \u2018Looked But Failed To See\u2019 (LBFTS) errors point to a common mechanism underlying these failures, whether the missed item was an unexpected gorilla, the clearly defined target of a visual search, or that simple typo. We argue that normal blindness is the by-product of the limited-capacity prediction engine that is our visual system. The processes that evolved to allow us to move through the world with ease are virtually guaranteed to cause us to miss some significant stimuli, especially in important tasks like driving and medical image perception.

Posted in: Psycho-physiological bases of engineering , Tagged:

On the existence of multiple fundamental “languages” in the brain that use discrete symbols and a few basic structures

November 3, 2023 09:03 ,

Stanislas Dehaene, Fosca Al Roumi, Yair Lakretz, Samuel Planton, Mathias Sabl�-Meyer, Symbols and mental programs: a hypothesis about human singularity, Trends in Cognitive Sciences, Volume 26, Issue 9, 2022, Pages 751-766 DOI: 10.1016/j.tics.2022.06.010.

Natural language is often seen as the single factor that explains the cognitive singularity of the human species. Instead, we propose that humans possess multiple internal languages of thought, akin to computer languages, which encode and compress structures in various domains (mathematics, music, shape\u2026). These languages rely on cortical circuits distinct from classical language areas. Each is characterized by: (i) the discretization of a domain using a small set of symbols, and (ii) their recursive composition into mental programs that encode nested repetitions with variations. In various tasks of elementary shape or sequence perception, minimum description length in the proposed languages captures human behavior and brain activity, whereas non-human primate data are captured by simpler nonsymbolic models. Our research argues in favor of discrete symbolic models of human thought.

Posted in: Psycho-physiological bases of engineering , Tagged: ,

Survey on RL applied to cyber-security

October 27, 2023 06:19 ,

Amrin Maria Khan Adawadkar, Nilima Kulkarni, Cyber-security and reinforcement learning \u2014 A brief survey, Engineering Applications of Artificial Intelligence, Volume 114, 2022, DOI: 10.1016/j.engappai.2022.105116.

This paper presents a comprehensive literature review on Reinforcement Learning (RL) techniques used in Intrusion Detection Systems (IDS), Intrusion Prevention Systems (IPS), Internet of Things (IoT) and Identity and Access Management (IAM). This study reviews scientific documents such as journals and articles, from 2010 to 2021, extracted from the Science Direct, ACM, IEEEXplore, and Springer database. Most of the research articles published in 2020 and 2021, for cybersecurity and RL are for IDS classifiers and resource optimization in IoTs. Some datasets used for training RL-based IDS algorithms are NSL-KDD, CICIDS, and AWID. There are few datasets and publications for IAM. The few that exist focus on the physical layer authentication. The current state of the art lacks standard evaluation criteria, however, we have identified parameters like detection rate, precision, and accuracy which can be used to compare the algorithms employing RL. This paper is suitable for new researchers, students, and beginners in the field of RL who want to learn about the field and identify problem areas.

Posted in: Computer vision , Tagged: ,

Hierarchical RL with diverse methods integrated in the framework

October 27, 2023 06:12 ,

Ye Zhou, Hann Woei Ho, Online robot guidance and navigation in non-stationary environment with hybrid Hierarchical Reinforcement Learning, Engineering Applications of Artificial Intelligence, Volume 114, 2022 DOI: 10.1016/j.engappai.2022.105152.

Hierarchical Reinforcement Learning (HRL) provides an option to solve complex guidance and navigation problems with high-dimensional spaces, multiple objectives, and a large number of states and actions. The current HRL methods often use the same or similar reinforcement learning methods within one application so that multiple objectives can be easily combined. Since there is not a single learning method that can benefit all targets, hybrid Hierarchical Reinforcement Learning (hHRL) was proposed to use various methods to optimize the learning with different types of information and objectives in one application. The previous hHRL method, however, requires manual task-specific designs, which involves engineers\u2019 preferences and may impede its transfer learning ability. This paper, therefore, proposes a systematic online guidance and navigation method under the framework of hHRL, which generalizes training samples with a function approximator, decomposes the state space automatically, and thus does not require task-specific designs. The simulation results indicate that the proposed method is superior to the previous hHRL method, which requires manual decomposition, in terms of the convergence rate and the learnt policy. It is also shown that this method is generally applicable to non-stationary environments changing over episodes and over time without the loss of efficiency even with noisy state information.

Posted in: Applications of reinforcement learning to robots , Tagged: ,

Unexpected consequences of training smarthome systems with reinforcement learning: effects on human behaviours

October 27, 2023 06:02 ,

S. Suman, A. Etemad and F. Rivest, TPotential Impacts of Smart Homes on Human Behavior: A Reinforcement Learning Approach, IEEE Transactions on Artificial Intelligence, vol. 3, no. 4, pp. 567-580, Aug. 2022 DOI: 10.1109/TAI.2021.3127483.

Smart homes are becoming increasingly popular as a result of advances in machine learning and cloud computing. Devices, such as smart thermostats and speakers, are now capable of learning from user feedback and adaptively adjust their settings to human preferences. Nonetheless, these devices might in turn impact human behavior. To investigate the potential impacts of smart homes on human behavior, we simulate a series of hierarchical-reinforcement learning-based human models capable of performing various activities\u2014namely, setting temperature and humidity for thermal comfort inside a Q-Learning-based smart home model. We then investigate the possibility of the human models\u2019 behaviors being altered as a result of the smart home and the human model adapting to one another. For our human model, the activities are based on hierarchical-reinforcement learning. This allows the human to learn how long it must continue a given activity and decide when to leave it. We then integrate our human model in the environment along with the smart home model and perform rigorous experiments considering various scenarios involving a model of a single human and models of two different humans with the smart home. Our experiments show that with the smart home, the human model can exhibit unexpected behaviors such as frequent changing of activities and an increase in the time required to modify the thermal preferences. With two human models, we interestingly observe that certain combinations of models result in normal behaviors, while other combinations exhibit the same unexpected behaviors as those observed from the single human experiment.

Posted in: Psycho-physiological bases of engineering , Tagged: , ,

Human+machine sequential decision making

October 27, 2023 05:59 ,

Q. Zhang, Y. Kang, Y. -B. Zhao, P. Li and S. You, Traded Control of Human\u2013Machine Systems for Sequential Decision-Making Based on Reinforcement Learning, IEEE Transactions on Artificial Intelligence, vol. 3, no. 4, pp. 553-566, Aug. 2022 DOI: 10.1109/TAI.2021.3127857.

Sequential decision-making (SDM) is a common type of decision-making problem with sequential and multistage characteristics. Among them, the learning and updating of policy are the main challenges in solving SDM problems. Unlike previous machine autonomy driven by artificial intelligence alone, we improve the control performance of SDM tasks by combining human intelligence and machine intelligence. Specifically, this article presents a paradigm of a human\u2013machine traded control systems based on reinforcement learning methods to optimize the solution process of sequential decision problems. By designing the idea of autonomous boundary and credibility assessment, we enable humans and machines at the decision-making level of the systems to collaborate more effectively. And the arbitration in the human\u2013machine traded control systems introduces the Bayesian neural network and the dropout mechanism to consider the uncertainty and security constraints. Finally, experiments involving machine traded control, human traded control were implemented. The preliminary experimental results of this article show that our traded control method improves decision-making performance and verifies the effectiveness for SDM problems.

Posted in: Human-robot interaction , Tagged: ,