SKELTER: unsupervised skeleton action denoising and recognition using transformers - Paoletti, Giancarlo; Beyan, Cigdem; Del Bue, Alessio (Frontiers in Computer Science, 2023)
Unsupervised Human Action Recognition (U-HAR) methods currently leverage large-scale datasets of human poses to solve this challenging problem. As most of the approaches are dedicated to reaching the best recognition accuracies, no attention has been put into analysing the resilience of such methods given perturbed data, a likely occurrence in real in-the-wild testing scenarios. Our first contribution is to systematically validate the decrease in performance of current U-HAR state of the art using perturbed or altered data (e.g., obtained by removing some skeletal joints, rotating the entire pose, injecting geometrical aberrations, etc.). Then, we propose a novel framework based on a transformer encoder-decoder with remarkable de-noising capabilities to counter such perturbations effectively. Moreover, we also present additional losses to have robust representations against rotation variances and to provide temporal motion consistency. Our model, SKELTER, shows limited drops in performance when skeleton noise is present compared to previous approaches, favouring its use in challenging in-the-wild settings.