Category Archives: Human-robot Interaction

A model of the psychomotor behaviour of humans intended to be useful for integration with robots

Stephen Fox, Adrian Kotelba, Ilari Marstio, Jari Montonen, Aligning human psychomotor characteristics with robots, exoskeletons and augmented reality, Robotics and Computer-Integrated Manufacturing, Volume 63, 2020, DOI: 10.1016/j.rcim.2019.101922.

In previous production literature, the uncertainty of human behaviour has been recognized as a source of productivity, quality, and safety problems. However, fundamental reasons for the uncertainty of human behavior have received little analysis in the production literature. Furthermore, potential for these fundamental reasons to be aligned with production technologies in order to improve production performance has not been addressed. By contrast, in this paper, fundamental reasons for the uncertainty of human behaviour are explained through a model of psychomotor characteristics that encompasses physiology, past experiences, personality, gender, culture, emotion, reasoning, and biocybernetics. Through reference to 10 action research cases, the formal model is applied to provide guidelines for planning production work that includes robots, exoskeletons, and augmented reality.

Symbol grounding through neural networks

Shridhar M, Mittal D, Hsu D., INGRESS: Interactive visual grounding of referring expressions, The International Journal of Robotics Research. January 2020, DOI: 10.1177/0278364919897133.

This article presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects. The key question here is to ground referring expressions: understand expressions about objects and their relationships from image and natural language inputs. INGRESS allows unconstrained object categories and rich language expressions. Further, it asks questions to clarify ambiguous referring expressions interactively. To achieve these, we take the approach of grounding by generation and propose a two-stage neural-network model for grounding. The first stage uses a neural network to generate visual descriptions of objects, compares them with the input language expressions, and identifies a set of candidate objects. The second stage uses another neural network to examine all pairwise relations between the candidates and infers the most likely referred objects. The same neural networks are used for both grounding and question generation for disambiguation. Experiments show that INGRESS outperformed a state-of-the-art method on the RefCOCO dataset and in robot experiments with humans. The INGRESS source code is available at

Sharing beliefs (pdfs) between human and robot

Rina Tse, Mark Campbell, Human–Robot Communications of Probabilistic Beliefs via a Dirichlet Process Mixture of Statements, IEEE Transactions on Robotics, vol. 34, no. 5, DOI: 10.1109/TRO.2018.2830360.

This paper presents a natural framework for information sharing in cooperative tasks involving humans and robots. In this framework, all information gathered over time by a human–robot team is exchanged and summarized in the form of a fused probability density function (pdf). An approach for an intelligent system to describe its belief pdfs in English expressions is presented. This belief expression generation is achieved through two goodness measures: semantic correctness and information preservation. In order to describe complex, multimodal belief pdfs, a Mixture of Statements (MoS) model is proposed such that optimal expressions can be generated through compositions of multiple statements. The model is further extended to a nonparametric Dirichlet process MoS generation, such that the optimal number of statements required for describing a given pdf is automatically determined. Results based on information loss, human collaborative task performances, and correctness rating scores suggest that the proposed method for generating belief expressions is an effective approach for communicating probabilistic information between robots and humans.