Shape detection beyond the visual field using a visual-to-auditory sensory augmentation device - Shvadron, Shira; Snir, Adi; Maimon, Amber; Yizhar, Or; Harel, Sapir; Poradosu, Keinan; Amedi, Amir (Frontiers in Human Neuroscience, 2023)
Current advancements in both technology and science allow us to manipulate our sensory modalities in new and unexpected ways. In the present study, we explore the potential of expanding what we perceive through our natural senses by utilizing a visual-to-auditory sensory substitution device, the EyeMusic, an algorithm that converts images to sound. The EyeMusic was initially developed to allow blind individuals to create a spatial representation of information arriving from a video feed at a slow sampling rate. Here we aimed to use the EyeMusic for the blind areas of sighted individuals. We use it in this initial proof of concept study to test the ability of sighted subjects to combine visual information with surrounding auditory sonification representing visual information. Participants in this study were tasked with recognizing and adequately placing the stimuli, using sound to represent the areas outside the standard human visual field. As such, the participants were asked to report shapes’ identities as well as their spatial orientation (Front/Right/Back/Left), requiring combined visual (90° frontal) and auditory input (the remaining 270°) for the successful performance of the task (content in both vision and audition was presented in a sweeping clockwise motion around the participant). We found that participants were successful at a highly above-chance level after a brief one-hour-long session of online training and one on-site training session of an average of 20 minutes. They could even draw a 2D representation of this image in some cases. Participants could also generalize, recognizing new shapes they were not explicitly trained on. Our findings provide an initial proof of concept indicating that sensory augmentation devices and techniques can potentially be used in combination with natural sensory information in order to expand the natural fields of sensory perception.