Category Decoding of Visual Stimuli From Human Brain Activity Using a Bidirectional Recurrent Neural Network to Simulate Bidirectional Information Flows in Human Visual Cortices - Qiao, Kai; Chen, Jian; Wang, Linyuan; Zhang, Chi; Zeng, Lei; Tong, Li; Yan, Bin (Frontiers in Neuroscience, 2019)
Recently, visual encoding and decoding based on functional Magnetic Resonance Imaging (fMRI) have made many great achievements with the rapid development of deep network computation. In spite of hierarchically similar representations of deep network and human vision, the visual information flows from primary visual cortices to high visual cortices, and conversely, from high visual cortices to primary visual cortices respectively based on the bottom-up manner and top-down manner. Inspired by the bidirectional information flows, we proposed the bidirectional recurrent neural network (BRNN) based method to decode the category from fMRI data. The forward and backward directions in the BRNN module characterized the bottom-up and top-down manners respectively. The proposed method regarded selected voxels of each visual cortex region (V1, V2, V3, V4 and LO) as one node in the sequence that feed into the BRNN module, and combined the output of BRNN module to decode the category with the subsequent fully connected layer. In this way, not only hierarchical information representations but also bidirectional information flows in human visual cortices can be efficiently used. Experiment results demonstrated that our method improved the accuracy on the three-level category decoding task than other methods, which implicitly validated the hierarchical and bidirectional human visual representations. Through comparison, we also analyzed and concluded that the category representations of human visual cortices were hierarchical, distributed, complementary and correlative.