RETRACTED: Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction - Niu, Ke; Zhang, Ke; Peng, Xueping; Pan, Yijie; Xiao, Naian (Frontiers in Molecular Biosciences, 2023)
In intensive care units (ICUs), mortality prediction is performed by combining information from these two sources of ICU patients by monitoring patient health. Respectively, time series data generated from each patient admission to the ICU and clinical records consisting of physician diagnostic summaries. However, existing mortality prediction studies mainly cascade the multimodal features of time series data and clinical records for prediction, ignoring the cross-modal correlation between the underlying features in different modal data. To address these issues, we propose a multimodal fusion model for mortality prediction that jointly models patients' time-series data as well as clinical records. We apply a fine-tuned Bert model (Bio-Bert) to the patient's clinical record to generate a holistic embedding of the text part, which is then combined with the output of an LSTM model encoding the patient's time-series data to extract valid features. The global contextual information of each modal data is extracted using the fusion module to capture the correlation between different modal data. Finally the multimodal fusion matrix is used for mortality prediction. We use a real dataset containing 18904 ICU patients to train and evaluate our model, and the research results show that the representations obtained by the model can achieve better prediction accuracy compared to the baseline approach.