The training set comprised a total of 565 patients diagnosed with hypercapnic respiratory failure who were admitted to the First People’s Hospital of Yancheng from October 2020 to September 2021. The external validation dataset consisted of 132 patients with hypercapnic respiratory failure hospitalized at the People’s Hospital of Jiangsu Province between October 2021 and December 2021. The primary objective of this data partitioning was to assess the model’s generalizability across different institutions, patient populations, and time periods. The training dataset included a relatively large number of patients primarily sourced from a regional hospital, while the external validation dataset originated from a higher-level medical institution, thereby simulating the diversity of real-world application scenarios. Although the current datasets are derived from two hospitals, we have selected data from different levels of care and across continuous time periods, which enhances the robustness of the model’s generalization. To demonstrate consistent performance across diverse datasets, we also evaluated the model’s generalizability.

The criteria for research patients with HRF were as follows: Arterial oxygen partial pressure (PaO2) was less than 8.0 kPa (60 mmHg) and arterial carbon dioxide partial pressure (PaCO2) was greater than 6.0 kPa (45 mmHg) based on blood gas analysis (16).

Patients who had incomplete clinical data, were under the age of 18, experienced death during hospitalization, suffered from trauma or malignancy (including hematological malignancy), or were pregnant were excluded from the study. Following a comprehensive understanding and explanation of the study procedure, all participants provided written informed consent.

The following clinical data were collected within 24 h of admission: demographic information, clinical manifestations, comorbidities, various scoring systems upon admission, laboratory test results, etc. The follow-up indicators involved monitoring the post-discharge survival time of patients diagnosed with HRF for a period of 2 years. The patient’s tracking process was conducted by telephone interviews and the further verification of their condition was used by the hospital system.

The current project follows the principles of the Declaration of Helsinki. The study was approved by the ethics committees of the First People’s Hospital of Yancheng (No. 2020-K062) and the People’s Hospital of Jiangsu Province (No. 2021-SR-346). The participants at both hospitals were required to provide informed written consent in order to participate in the clinical study.

The modeling data was utilized for model construction and validation, while the external validation data was employed for subsequent model verification. The flowchart of this experiment for a multi-center prospective cohort study was depicted in Figure 1. Firstly, variable screening was performed using LASSO regression (Least Absolute Shrinkage and Selection Operator), followed by the construction of a multivariate CoxPH model based on the selected variables. Simultaneously, two models (RSF and DeepSurv) were constructed using the selected variables, and hyperparameters were optimized in the training set using mesh search method to determine the optimal values. The performance of the three models was subsequently assessed in both the modeling data set and the external validation data set.

CoxPH, as a classical parametric method, offers high interpretability and robustness. RSF employs non-parametric methods to capture nonlinear relationships and variable interactions, demonstrating greater resilience to data quality issues and multicollinearity. DeepSurv leverages deep learning techniques to handle high-dimensional and complex data, excelling in modeling nonlinear features and variable interactions. By integrating these three models, we can accommodate diverse data characteristics, achieve comprehensive and robust analysis both theoretically and practically, and provide multidimensional support for our research objectives.

The following evaluation indicators were employed: the conformance index (C-index) and the area under the receiver operating characteristic curve (at 6, 12, 18, and 24 months) was utilized to evaluate the discriminative ability of each model. The calibration of the model was assessed using Brier Score (at 6, 12, 18, and 24 months). Decision curve analysis (at 6, 12, 18, and 24 months) was utilized to calculate the clinical net benefit of each model.

The C-index reflects the ability of the model to distinguish between high and low risk individuals, with values ranging from 0.5 (nearly random) to 1.0 (completely accurate). The higher the value, the more accurately the model can distinguish between high and low risk individuals (17). The Brier Score assesses the accuracy and reliability of probabilistic predictions made by a model, with values ranging from 0 to 1. A lower score indicates a smaller prediction error, thereby reflecting greater accuracy and consistency in the model’s forecasts (18).

Time-dependent ROC analysis serves as a robust tool for assessing the predictive accuracy of survival models across various time points. By computing the sensitivity and specificity at each time point, a time-dependent ROC curve is generated, and the AUC (area under the curve) is calculated to quantify the model’s predictive performance. This method initially estimates an individual’s survival probability or risk score using a survival model. Subsequently, it constructs a time-dependent confusion matrix based on varying thresholds to evaluate the model’s classification performance at specific time intervals (19, 20). For right-censored data, the inverse probability weighting (IPW) method is employed to adjust for censoring bias, thereby ensuring the precision of AUC calculations. Through repeated analyses at multiple time points, the temporal changes in the model’s predictive power can be dynamically monitored and utilized for comparing different survival models (21, 22).

The risk score for each patient can be calculated using the optimal model derived from comparing various models. Patients were stratified based on their risk score, and survival analysis was conducted using Kaplan-Meier curves with differences compared using the log-rank test.

The SHAP plots served as a valuable tool for interpreting machine learning models. Within these plots, the length of the horizontal axis corresponding to each variable signified its contribution to the outcome, while the color of the dots represented the magnitude of that contribution.

The individual predictions comprised survival probability plots and local SHAP plots, which offered distinct perspectives on survival expectations and risk factors, respectively. The survival probability of each individual was calculated using non-parametric estimation. The local SHAP plot was an interpretive representation that showed the individual-specific contribution of a variable in a SHAP plot. The establishment of an association between risk factors and individual outcomes was facilitated based on the plots. Furthermore, we have developed a web-based prediction tool utilizing the Shiny framework to estimate patient survival probabilities at specified time points.

The categorical variables in both the modeling data set and the external validation data set were presented using frequency and percentage, and analyzed using either chi-square test or Fisher exact test. Continuous variables conforming a normal distribution were represented by their mean and standard deviation, and analyzed using the t-test. Continuous variables not conforming to a normal distribution were represented by their median and quartile, and analyzed using the Mann-Whitney U test. A p-value < 0.05 in bilateral test was considered statistically significant. The mlr3proba package for R (version 0.4.13) was utilized to construct three survival machine learning models, wherein the DeepSurv model relied on the Pycox module. The survival machine learning model was explained using Python’s SHAP module (version 0.37.0). The CatPredi package in the R language was utilized for discretizing risk scores into layers.

The study included a total of 697 patients with HRF, and Table 1 presented the demographic and clinical information comparing the modeling data set to the external validation data set.

The study incorporated a total of 85 variables. Utilizing the training dataset, we employed the LASSO regression (Least Absolute Shrinkage and Selection Operator) method for variable selection. The optimal regularization parameter λ was determined through 10-fold cross-validation, with the model corresponding to the minimum value being selected as the final model (Figure 2). By incorporating an L1 norm regularization penalty term into the model, LASSO regression achieved both variable selection and coefficient shrinkage. This approach not only mitigated overfitting and enhanced the model’s generalization capability but also addressed multicollinearity issues. Through this method, 17 variables with the highest explanatory power for the research objective were ultimately selected, including: Gender, ICU admission, Hypertension disease, ILD, BMI, Braden score, mMRC score, Padua score, Carboxyhemoglobin, Arterial hematocrit, Lymphocytes percentage, Hemoglobin, GGT, Albumin, Cholinesterase, Urea nitrogen, LDH.

RSF model: The 17 variables selected by LASSO were used for modeling. Grid search was conducted, and a 10-fold cross-validation was performed to determine the optimal combination of hyperparameters for the model, including ntree set at 900, mtry set at 3, nodesize set at 18. The remaining hyperparameters were kept as default values.

CoxPH model: A multivariate Cox regression analysis was conducted using 17 variables selected by LASSO. The analysis identified ICU admission, ILD, BMI, mMRC score, Carboxyhemoglobin, and Urea nitrogen as significant prognostic factors in patients with HRF. The results of the multivariate Cox regression analysis were shown in Table 2.

DeepSurv model: The 17 variables selected by LASSO were used for modeling. Grid search and 10-fold cross-validation were used to determine the optimal combination of hyperparameters of the model, including alpha set at 0.5, learning_rate set at 0.005c, num_nodes set at 10. The remaining hyperparameters were kept as default values.

The C-index of each model was compared in both the modeling data set and the external validation set to compare their respective discriminatory abilities. The C indices of the RSF model, CoxPH model, and DeepSurv model (hereinafter referred to as the “three models”) on the modeling dataset were 0.792, 0.699, and 0.618, respectively. On the external validation set, these indices were 0.693, 0.681, and 0.532 for each respective model. These results indicated that the RSF model exhibited superior discriminative ability.

The comparison of the discrimination among the three models at 6, 12, 18, and 24 months on the modeling data set revealed that the RSF model exhibited superior discrimination (Figure 3). The external validation set demonstrated that the RSF model exhibited superior discrimination at 6, 12, and 18 months compared to the other models, while its discrimination at 24 months was comparable to that of the Cox regression model (Figure 4).

On the modeling data set, the Brier Score was utilized to assess the model’s predictive consistency at four time points: 6, 12, 18, and 24 months. The results showed that the Brier Score of the RSF model was the best and was less than 0.25 at each time point. Similarly, on the external validation set, the Brier Score was utilized to evaluate the model’s predictive consistency at 6, 12, 18, and 24 months. The results demonstrated that once again, the RSF model outperformed other models by maintaining a Brier Score below 0.25 for each respective time point. The above information was shown in Table 3.

On the modeling data set, a DCA comparison was conducted for the three models at 6, 12, 18, and 24 months. The results revealed that the RSF model exhibited superior clinical net benefit (Figure 5). The RSF model also demonstrated superior clinical net benefit at 6, 12, and 18 months on the external validation set, while being comparable to Cox regression at 24 months (Figure 6).

In conclusion, the performance of the random survival forest (RSF) model in survival analysis demonstrated significant advantages. Specifically, regarding its differentiation ability, the C-index of the RSF model outperformed both the Cox regression model and the DeepSurv model across both the modeling data and external validation sets. This indicated that the RSF model possesses superior individualized prognostic differentiation capabilities. Regarding prediction consistency, the Brier Score of the RSF model demonstrated lower error rates across multiple time points, thereby further validating the accuracy and stability of its predictive performance. Simultaneously, clinical net benefit analysis demonstrated that the RSF model exhibited superior benefits during short-term follow-up and could more effectively support clinical decision-making. From a clinical perspective, the high predictive accuracy and interpretability of the RSF model not only enhance the precision of patient risk assessment but also provide a robust tool for personalized prognostic prediction and treatment optimization.

Additionally, the RSF model was visually elucidated. In the SHAP diagram, the variables in the model were presented in a descending order of importance (Figure 7). Amongst the initial five variables, Cholinesterase emerged as the most crucial, succeeded by Urea nitrogen, ICU admission, BMI, and Albumin.

The patients in the validation set were categorized into three groups based on their risk scores: high-risk group (risk score > 1.06), medium-risk group (0.24 ≤ risk score ≤ 1.06), and low-risk group (risk score < 0.24). The Kaplan-Meier analysis and logarithmic rank test results depicted in Figure 8 demonstrated statistically significant differences among the high-risk, medium-risk, and low-risk groups as a whole.

The web-based prediction tool was developed utilizing the random survival forest (RSF) model. This tool, implemented via Shiny,¹ offered a user-friendly interface, real-time predictive capabilities, and comprehensive visualization features. It served as an efficient and precise solution for individualized risk assessment and survival prediction, specifically tailored for clinical practitioners. By dynamically generating patient survival probability curves and estimating survival rates at specific time points, this tool offered robust quantitative support for clinical decision-making. It significantly enhanced the operability and adaptability of the model in practical applications, thereby providing crucial technical support for the individualized management of HRF.

Three patients were randomly selected and numbered sequentially to demonstrate individual prognosis. The individual predicted survival rate is illustrated in Figure 9A. It can be observed that the third patient exhibited a relatively favorable survival rate, whereas the first patient exhibited a relatively unfavorable survival rate. The local SHAP plot elucidated the prognosis for each patient in terms of the contribution of the variables, wherein red stripes represented risk factors associated with poor prognosis, while blue stripes indicated relative protective factors.

Patient 1: As shown in Figure 9B, the local SHAP plot revealed that the top five most important variables, namely ICU admission, Cholinesterase, Albumin, LDH, and Lymphocytes percentage, were all identified as risk factors associated with poor prognosis.

Patient 2: As shown in Figure 9C, the local SHAP plot revealed that ICU admission, LDH, Hemoglobin hematocrit and Arterial hematocrit were the risk factors with poor prognosis among the top five most important variables, while Cholinesterase was the protective factor.

Patient 3: As shown in Figure 9D, the local SHAP plot revealed that ICU admission was the risk factor with poor prognosis among the top five most important variables, while Urea nitrogen, Cholinesterase, Braden score and LDH were the protective factors.