Spatio-temporal categorization for first-person-view videos using a convolutional variational autoencoder and Gaussian processes - Nagano, Masatoshi; Nakamura, Tomoaki; Nagai, Takayuki; Mochihashi, Daichi; Kobayashi, Ichiro (Frontiers in Robotics and AI, 2022)
Humans perceive continuous high-dimensional information by dividing and categorizing it into significant segments. We believe that this unsupervised segmentation capability is important for mobile robots to learn spatial knowledge. In this study, we propose HcVGH, a method that learns spatio-temporal categories by segmenting first-person-view (FPV) videos captured by mobile robots. HcVGH combines a convolutional variational autoencoder (cVAE) with HVGH, the product of our past work, which follows the hierarchical Dirichlet process-variational autoencoder-Gaussian process-hidden semi-Markov model comprising deep generative and statistical models. HcVGH not only segments a time series of vectors but also handles video data (i.e., image time series) in an unsupervised manner. In an experiment, we used FPV videos of agent in a maze and divided them into significant segment classifications. The results confirm that our proposed model stably estimates more accurate video segments than the baseline method, the hierarchical recurrent state space model. Furthermore, the learned parameters of HcVGH accurately represent the spatio-temporal structure of the maze in question.