American Sign Language Translation Using Wearable Inertial and Electromyography Sensors for Tracking Hand Movements and Facial Expressions - Gu, Yutong; Zheng, Chao; Todoh, Masahiro; Zha, Fusheng (Frontiers in Neuroscience, 2022)
A sign language translation system can break the communication barrier between hearing-impaired people and others. In this paper, a novel American sign language translation method based on wearable sensors is proposed. We leverage inertial sensors to capture signs and surface electromyography sensors to detect facial expressions. We apply convolutional neural network to extract features from input signals. Then, long short-term memory and transformer models are exploited to achieve end-to-end translation from input signals to text sentences. We evaluate two models on 40 American sign language sentences strictly following the rules of grammar. Word error rate and sentence error rate are utilized as evaluation standard. The long short-term memory model can translate sentences in testing dataset with 7.74% word error rate and 9.17% sentence error rate. The transformer model performs much better than long short-term memory by achieving 4.22% word error rate and 4.72% sentence error rate. Both models are suitable for sign language translation with high accuracy.