The retrospective cohort analysis was performed at Yonsei Cancer Center, Seoul, Korea. Based on the EMR review, we identified patients who were diagnosed with rectal cancer between April 2012 and October 2019. The EMR data, including the rectal magnetic resonance imaging (MRI) reports, sexes, ages, and carcinoembryonic antigen (CEA) levels, determined by electrochemiluminescence immunoassay, of the rectal cancer patients, were obtained. A total of 25 radiologists reported MRI reading during that period. Patients without rectal MRI reports were excluded. The study was reviewed and approved by the institutional review and ethics board of the Severance Hospital, Seoul, Korea (IRB no. 4-2020-1003). The requirement of obtaining informed consent was waived owing to the retrospective nature of the study.

The primary endpoint of this study was the concordance of the patient risk predicted by the deep-transfer-learning-based survival prediction model and the actual overall patient survival. The proposed model predicted patient survival based on four steps (model structure shown in Figure 1 ). The first step was to convert the input radiological report into an embedding vector that corresponded to each word with a bidirectional encoder representation from transformers (BERT) language model (16), which, among other recent language models, has demonstrated state-of-the-art performance in many fields. The second step was to aggregate the word-embedding vectors into single-feature vectors to represent the radiological reports using an attention technique (17–19) for weighting the more important words in the report. The third step was the serial analysis of multiple radiological reports from a single patient to obtain a patient embedding vector using the gated rectified unit (20) among all RNN implementations. Finally, patient survival was predicted using a Cox-PH (21) model with the patient embedding vector. Detailed descriptions of the loss function and hyperparameters used in the deep-transfer learning are provided in Supplementary Material .

To extract high- and low-risk features from radiology reports, we obtained N-grams, a continuous sequence of N items in a text or sequence, from the radiology reports for each risk group. Then, N-grams with statistically significant differences in the frequency of appearance for each risk group were selected, and we clustered the selected N-grams using edit distance (22) and determined the representative features from each cluster. See Supplementary Material for details.

To investigate what determines the survival prediction of the deep-transfer learning model, the patient embedding vectors were extracted from the last hidden layer of the model using the first radiological report of the patient. The embedding vectors were then reduced to two-dimensional vectors using the t-stochastic neighbor embedding (t-SNE) algorithm (23) for visualization. The p-values were calculated using the log-rank test (24) to compare survival between patient groups stratified by the appearance of the terms.

Based on the patient risk, the N-year survival prediction performance was also evaluated using the area under the receiver operating characteristic (AUROC) curve. The patients in the test set were assigned binarized labels with 1-, 2-, 4-, and 5-year survival thresholds. The binarized labels for the patients in the test set were assigned to be positive for patients who survived longer than a certain threshold and negative for patients who did not.

The performance metrics that were used to evaluate the survival prediction of the deep-transfer-learning model were the C-index (25) and the p-value of the log-rank test. To evaluate the association between the patient risk and actual survival time, the Spearman correlation coefficient (R_s ) (26) was used. A p-value < 0.05 was considered statistically significant. See Supplementary Material for details.

In total, the EMR data of 7,402 patients diagnosed with rectal cancer from April 2012 to October 2019 were reviewed, and data of 4,338 patients with available rectal MRI radiological reports were analyzed. Approximately 40% of the patients had two or more serial MRI reports available. The survival cut-off date was October 28, 2019 (median follow-up duration, 49 months). Among the patients, 3,470 were assigned to the training dataset and 868 were assigned to the test dataset ( Supplementary Figure 1 ). To maximize the effectiveness of the unstructured data, the clinical data, including the sex, age, and CEA levels, were not used. Only survival or death events were used for the deep-transfer-learning model implementation, which were well balanced between the training and test sets ( Table 1 ).

We developed a novel deep-transfer-learning model to predict patient survival using unstructured serial MRI radiological reports from rectal cancer patients. Firstly, the radiological report of a patient was represented in the form of word-embedded hidden vectors by a pre-trained language model that comprehends the input text as a sequence of words. For the pre-trained language model, we used ClinicalBERT (27), which employs transferred knowledge extracted from public clinical notes. The hidden vectors were then aggregated to construct the feature vector of the patient, which was used as input to the Cox-PH model, which predicts the patient survival risk. For an individual patient with multiple serial rectal MRI radiological reports available, the risk status was predicted by an RNN that combined all the feature vectors obtained from all the available radiological reports.

To evaluate the performance of the deep-transfer-learning model, three comparative experiments were conducted. Firstly, survival predictions obtained from four different language models were compared. An un-pre-trained BERT and three pre-trained BERT language models (original BERT (16), BioBERT (28), and ClinicalBERT (27)) were compared through five-fold cross-validation with the training dataset and external validation using a separate test set (see Supplementary Material and Supplementary Table 1 ). The model performance was evaluated using the concordance index (C-index) (25), which is widely used to evaluate the performance of survival prediction models. C-index shows a higher value when the higher survival probability does the model predict for the long-time survived patients. Among the three language models, ClinicalBERT yielded the best survival prediction performance.

Subsequently, we assessed whether survival prediction was improved when multiple serial MRI radiological reports from the patient were utilized. The deep-transfer-learning model predicted the patient survival with serial radiological reports using the RNN, as described earlier. Compared with the single-time-point model (using only the first radiological report from the patient), the serial model (using multiple serial radiological reports from the patient) exhibited an improved C-index for the entire test set (C-index of 0.595 vs. 0.579, p-value < 4.9×10⁻², Table 2 ). In particular, when two serial radiological reports were used, the survival prediction of the patient was improved in the serial model case (C-index of 0.667) compared with those of patients with single radiological reports (C-index of 0.626). These results imply that the serial model has a better prediction of survival when interpreting subsequent MRI reports that occur following the course of treatment based on the patient’s first MRI report.

We divided the patients into three subgroups (high-, intermediate-, and low-risk) based on the predicted risk for each patient. Then, the deep-transfer-learning-based survival prediction model successfully stratified patients into the high-risk group (defined as patients with risks in the top 33rd percentile according to the deep-transfer-learning model) or low-risk group (patients with risks in the bottom 33rd percentile) (log-rank p-value < 2.0×10⁻³, Figures 2A, B ).

In addition, two experiments were conducted to compare the effect of the risk prediction obtained from the tumor marker CEA, which is known to predict the survival outcome of colorectal cancer patients (29), on survival prediction with those of the radiographic reports. Firstly, we visualized how the CEA and risks predicted by our model correlated with the actual survival time using the Spearman correlation coefficient (R_s ) (26). The scatter plot results revealed that the predicted risk (R_s = −0.590) had a higher correlation with the actual survival time compared with the CEA (R_s = −0.195) ( Figures 2C, D ). Secondly, we compared the effectiveness of CEA and the risks predicted by our model in forecasting N-year survival using receiver operating characteristic (ROC) curves. Use of the risk obtained through the deep-transfer-learning model yielded higher AUROC values than that of CEA in predicting N-year survival ( Figure 2E ). Additionally, we obtained ROC curves for patients with more than one MRI report ( Supplementary Figure 5 ) and found higher AUROC values than the curves for whole patients in most cases. This result means our model performed better when a patient had serial radiological reports.

To obtain deeper clinical insights for utilization of the proposed deep-transfer-learning model, we visualized the embedding vectors of the patients and investigated the hidden knowledge that the deep-transfer-learning model learned from the training data. We considered the hidden vector of the last layer of the model as the patient embedding vector and obtained the scatter plot visualization via dimensionality reduction with the t-SNE algorithm ( Figure 3 ). When we projected the actual patient survival on the same scatterplot, we identified the distribution of clusters of patients with poor versus favorable survival ( Figure 3A ). To identify which clinicopathologic features were related to high-risk patients, we obtained sequences of words (N-grams) that were prevalent among MRI reports of a high-risk group compared with those of the low-risk group. Then we extracted representative high-risk term features based on clustering the sequence of words. In the same way, the representative low-risk term features were extracted. We found the extracted term features were closely related to the patients’ prognosis and clinically acceptable ( Supplementary Figure 2 ). Then we visualized these term features on the scatterplot. Terms that were clinically related to worse prognoses, such as “T3,” “M-rectum,” and “with mesorectal fat infiltration,” were highly prevalent among patients with worse survival [p-value < 0.05 between patient groups stratified by the appearance of the terms ( Figures 3B–D )]. In addition, other terms associated with negative prognoses, such as “enlarged lymph node,” “CRM threatening,” and “suspicious regional lymph node metastases,” were prevalent among patients with worse prognoses. Moreover, a positive prognostic term, “No evidence of significant lymph node enlargements in both pelvic side walls,” was prevalent among patients with favorable survival ( Supplementary Figure 3 ).

Using the trained survival risk prediction model, we produced a survival graph for the radiological reports of a new patient. To obtain reliable survival information, the survival graph of the patient was generated by aggregating the survival information of other patients in the training dataset who were the most similar to the given patient in terms of the predicted risk. After predicting the survival risk value of the new patient using the radiological reports of the patient, the Kaplan–Meier survival curve (30, 31) was plotted using the survival data from 5% of patients in the training dataset with the most similar risk values in the deep-transfer-learning model. Supplementary Figure 4 presents an example of the survival-graph generation of a new patient. The survival risks predicted by the deep-transfer-learning model for three new rectal cancer patients with MRI radiological reports were plotted in a distribution graph, and Kaplan–Meier survival plots were drawn for each patient.