AUTHOR=Li Zhihao , Mu Yishan , Sun Zhenglong , Song Sifan , Su Jionglong , Zhang Jiaming 

TITLE=Intention Understanding in Human–Robot Interaction Based on Visual-NLP Semantics

JOURNAL=Frontiers in Neurorobotics

VOLUME=Volume 14 - 2020

YEAR=2021

URL=https://www.frontiersin.org/journals/neurorobotics/articles/10.3389/fnbot.2020.610139

DOI=10.3389/fnbot.2020.610139

ISSN=1662-5218

ABSTRACT=With the rapid development of robotic and AI technology in recent years, human-robot interaction has made 
great advancement, making a practical social impact. Verbal commands are one of the most direct and frequently used means for human-robot interaction. Currently such technology enable robots to execute pre-defined tasks based on simple, direct and explicit language instructions, e.g. certain keywords must be used and detected. However, that is not the natural way for humans to communicate. In this paper, we aim to propose a novel task-based framework to enable robots to comprehend human intentions from vague natural language instructions using visual semantics information. We utilize the robot camera view for instance segmentation and object identification, followed by a semantic analysis model to compute the degree of match between the visual and verbal information. Under a specified task framework, this approach is able to generate structured robot control languages which can be understood by the robot. To validate the efficacy of our method, experiments are carried out on a robotic arm to grasp objects that satisfy human intentions based on natural language instructions.