Crossmodal Pattern Discrimination in Humans and Robots: A Visuo-Tactile Case Study - Higgen, Focko L.; Ruppel, Philipp; Görner, Michael; Kerzel, Matthias; Hendrich, Norman; Feldheim, Jan; Wermter, Stefan; Zhang, Jianwei; Gerloff, Christian (Frontiers in Robotics and AI, 2020)
The quality of crossmodal perception hinges on two factors: The accuracy of the independent unimodal perception and the ability to integrate information from different sensory systems. In humans, the ability for cognitively demanding crossmodal perception diminishes from young to old age. Here, we propose a new approach to research to which degree the different factors contribute to crossmodal processing and the age-related decline by replicating a medical study on visuo-tactile crossmodal pattern discrimination utilizing state-of-the-art tactile sensing technology and artificial neural networks. We implemented two artificial neural network models focusing on the relevance of early integration of sensory information during the crossmodal processing stream as a mechanism proposed for efficient processing in the human brain. Approaching comparable unimodal performance for both modalities in the human participants as well as the artificial neural networks allowed us to compare crossmodal performance between and within the systems. Our data show that in an artificial system the integration of high-level unimodal features on earlier stages of the crossmodal processing stream outperforms the hierarchical integration of independent unimodal classifications. In comparison to humans, the artificial systems outperform older participants in the unimodal as well as the crossmodal condition. However, compared to younger participants, the artificial system performs worse in the crossmodal task at the estimated unimodal visual and tactile perception thresholds. Younger participants seem to employ more efficient crossmodal integration mechanisms than modelled in the proposed artificial neural networks. Our work demonstrates how collaborative research in neuroscience and embodied artificial neurocognitive models might help to derive hypotheses from the allied field. Our results indicate that empirically derived neurocognitive models can inform the design of future neurocomputational architectures. For crossmodal processing, human inspired early sensory integration on lower hierarchical levels seems to improve the performance of artificial neural networks.