Multimodal region-based behavioral modeling for suicide risk screening - Alghowinem, Sharifa; Zhang, Xiajie; Breazeal, Cynthia; Park, Hae Won (Frontiers in Computer Science, 2023)
Suicide is a leading cause of death around the world, interpolating a huge suffering to the families and communities of the individuals, who already longed suffered unbearable pain before ending their lives. Such pain and suffering are preventable with early screening and monitoring. However, current suicide risk identification relies on self-disclosure and/or the clinician's judgment. To fill this gap, we investigate prosody and nonverbal behavioral markers that are associated with different levels of suicide risks through a multimodal approach for suicide risk detection. Given the differences in the behavioral dynamics between subregions of facial expressions and body gestures in terms of timespans, we propose a novel region-based multimodal fusion. Using a newly collected video interview dataset of young Japanese at risk of suicide, we extracted engineered features and deep representations from the speech, regions of the face, regions of the body, as well as the overall combined regions. The results confirmed that behavioral dynamics differs between regions, where some regions benefit from a shorter timespans, while other regions benefit from longer ones. Therefore, a region-based multimodal behavioral is more informative in terms of behavioral markers and accounts for both subtle and strong behaviors. Comparing the region-based results, the handcrafted features from eyes, nose, mouth, and full body gestures showed superiority over other regions and other feature types, which validates previous findings in suicide behavioral markers in Western population. Our region-based multimodal results outperformed the single modality, reaching a sample-level accuracy of 96% compared with the highest single modality that reached sample-level accuracy of 80%. Even though multimodal is a powerful tool to enhance the model performance and its reliability, it is important to ensure through a careful selection that a strong behavioral modality (e.g. body movement) does not dominant another subtle modality (e.g. eye blink). Despite the small sample size, our unique dataset and the current results adds a new cultural dimension to the research on nonverbal markers of suicidal risks. Given a larger dataset, future work can be useful in helping psychiatrists with the assessment of suicide risk and could have several applications to identify those at risk.