Intention Understanding in Human–Robot Interaction Based on Visual-NLP Semantics - Li, Zhihao; Mu, Yishan; Sun, Zhenglong; Song, Sifan; Su, Jionglong; Zhang, Jiaming (Frontiers in Neurorobotics, 2021)
With the rapid development of robotic and AI technology in recent years, human-robot interaction has made great advancement, making a practical social impact. Verbal commands are one of the most direct and frequently used means for human-robot interaction. Currently such technology enable robots to execute pre-defined tasks based on simple, direct and explicit language instructions, e.g. certain keywords must be used and detected. However, that is not the natural way for humans to communicate. In this paper, we aim to propose a novel task-based framework to enable robots to comprehend human intentions from vague natural language instructions using visual semantics information. We utilize the robot camera view for instance segmentation and object identification, followed by a semantic analysis model to compute the degree of match between the visual and verbal information. Under a specified task framework, this approach is able to generate structured robot control languages which can be understood by the robot. To validate the efficacy of our method, experiments are carried out on a robotic arm to grasp objects that satisfy human intentions based on natural language instructions.