Compact Neural Architecture Designs by Tensor Representations - Su, Jiahao; Li, Jingling; Liu, Xiaoyu; Ranadive, Teresa; Coley, Christopher; Tuan, Tai-Ching; Huang, Furong (Frontiers in Artificial Intelligence, 2022)
We propose a framework of s (s), which extends existing linear layers on low-order tensors to multilinear operations on higher-order tensors. Learning s is challenging in general, which corresponds to tensor decomposition in a nonlinear setting. We address this problem by deriving backpropagation rules for s, with a novel suite of generalized tensor algebra. Our s have three advantages over existing networks: First, s naturally apply to higher-order data without flattening, which preserves their multi-dimensional structures. Second, compressing a pre-trained network into a results in a model with similar expressive power but fewer parameters. Finally, s interpret advanced compact designs of network architectures, such as bottleneck modules and interleaved group convolutions. Experiments on VGG, ResNet, and Wide-ResNet demonstrate that s outperform the state-of-the-art low-rank methods on a wide range of backbone networks and datasets.