This study validates prognostic models using data from an ambispective cohort of COVID-19 patients within the WHO Global Clinical Platform (11), to which Colombia contributed anonymized, individual-level data. The platform uses standardized ISARIC case report forms and a predefined analytic framework, with harmonized variable domains including demographics, baseline comorbidities, clinical presentation at admission, laboratory parameters, treatments, and clinical outcomes (11).

The cohort included patients aged 18 years or older who presented to the emergency department with suspected SARS-CoV-2 infection, later confirmed by RT-PCR or antigen testing, between March 2020 and September 2021. Data were collected from seven high-complexity hospitals: four in Bogotá, two in Barranquilla, and one in Cali.

Patients were included if they had confirmed SARS-CoV-2 infection and at least one COVID-19-related symptom (see Supplementary material M1). Exclusion criteria included transfer to another institution within 24 h or teleconsultation. To avoid duplicate observations, only the first emergency department visit per patient during the study period was included in the analysis. Cohort entry began at the time of the first emergency department consultation, and all patients were followed for up to 30 days after admission.

Model validation followed the TRIPOD (transparent reporting of a multivariable prediction model for individual prediction or diagnosis) guidelines (12, 13).

The variables were those originally included in each model. They encompassed demographic variables, comorbidities (modified Charlson Comorbidity Index) (14), vital signs at admission, clinical parameters (use of supplemental oxygen, and presence of confusion), and admission laboratory tests (urea, C-reactive protein). The recorded comorbidities were chronic cardiac disease, chronic respiratory disease (excluding asthma), chronic renal disease (estimated glomerular filtration rate ≤30), mild to severe liver disease, dementia, chronic neurological conditions, connective tissue disease, diabetes mellitus, HIV/AIDS, malignancy, and clinician-defined obesity.

The primary outcome was 30-day mortality, defined as death from any cause within 30 days from the emergency department visit. For discharged patients, survival status was confirmed via the Registro Único de Afiliados (RUAF) of the Colombian Ministry of Health (15). A blinded investigator verified death certificates. Additional data on hospitalization, intensive care unit (ICU) admission, and hospital stay length were also collected.

All seven participating hospitals used electronic health record (EHR) systems, which allowed standardized extraction of clinical variables using definitions aligned with the WHO/ISARIC case report forms (11, 16). Data entry followed the standardized procedures of the WHO Global Clinical Platform and was performed either through direct entry into the platform or via upload of a harmonized WHO Excel template. Laboratory data were transferred electronically from institutional laboratory information systems to the ISARIC database in all centers, ensuring uniform units of measurement and minimizing transcription errors. Data collection was performed by trained physician-investigators at each site. To ensure data quality and completeness, 10% of the sample was independently reviewed in duplicate. In addition, two investigators conducted monthly reviews of the Colombian National Public Health Surveillance System to verify consecutive patient inclusion and data accuracy.

Sample size calculation followed the Riley et al. methodology (17), using the 4C model (n = 20 parameters, C-statistic 0.78) and an estimated 11.2% mortality rate observed in a previous study based on the Colombian WHO Global Clinical Platform cohort (18). The minimum required sample size was 1,624 patients with 82 mortality events. Given our cohort of 7,973 patients and 901 deaths, the sample was sufficient for model validation.

Because the 4C model includes the largest number of predictors and represents the most statistically demanding validation scenario, meeting its sample size requirements also ensured adequate sample size for the validation of CURB-65, NEWS2, and qSOFA models.

Exploratory data analysis identified outliers, entry errors, and missing values, and adjustments were made as needed. No missing values were found in demographic or clinical variables. However, C-reactive protein and urea were missing in 2.0% (n = 159) and 2.7% (n = 216) of patients, respectively. The only model that incorporated both variables was the 4C, with 4.6% (n = 369) missing data. Since this percentage was below 5%, a complete-case analysis was conducted. There was no missing data for the outcome variable.

A descriptive analysis was conducted, summarizing continuous variables as medians and interquartile ranges (IQRs), and categorical variables as absolute and relative frequencies. The Kaplan–Meier method was used to estimate overall survival, with comparisons across predefined risk groups performed using log-rank tests for each model. Administrative censoring was applied at 30 days post-admission.

For model validation, mortality predictions were generated using CURB-65, CRB-65, NEWS2, qSOFA, and 4C models (see Supplementary material M2), applying the original scoring criteria (9, 19–21). Discrimination was assessed using the receiver operating characteristic (ROC) curves and the area under the curve (AUC) with a 95% confidence interval (95% CI), with AUC > 0.7 considered acceptable (13).

Calibration was evaluated using the GiViTI calibration belt. This method incorporates a confidence band to quantify prediction uncertainty and employs a likelihood ratio test, and to capture non-linear deviations from perfect calibration (22). Additionally, observed-to-expected (predicted) events were analyzed by risk group to provide a detailed performance assessment.

In addition, for the 4C and NEWS2 models, we conducted analyses to evaluate potential changes in model performance across key phases of the pandemic in Colombia. Discrimination and calibration were assessed separately for two relevant time periods: (a) before and after the implementation of corticosteroid therapy in routine clinical practice (July 2020) (23), and (b) before and after the launch of the national COVID-19 vaccination program in Colombia (April 2021).

For the 4C model, which provided intercept and regression coefficients (9). We calculated additional overall performance measures such as the Cox-Snell R-squared, Brier Score, and calibration metrics such as the calibration intercept and calibration slope.

Despite its good discriminatory performance, the 4C model exhibited poor calibration. As initial recalibration-in-the-large (adjusting only the intercept) was insufficient to correct this miscalibration. Therefore, logistic recalibration, involving adjustments to both the intercept and slope, was performed, resulting in improved discrimination and calibration. The choice of recalibration method was supported by a likelihood ratio test, which identified this approach as the most conservative and statistically appropriate (24, 25).

Finally, to assess the clinical utility of the 4C model, we conducted decision curve analysis (DCA) and evaluated net benefit, which balances intervention benefits against potential harms. To determine an appropriate risk threshold for death to guide hospitalization decisions for COVID-19, we selected a risk threshold range between 2 and 10% based on expert clinical consultation and previous prediction models (9, 19–21). The recalibrated 4C model was compared to the original model, and the strategies of hospitalizing all or none (10, 26).

Data analysis was performed in R v4.1 using the packages pmsampsize, predRupdate, pROC, givitiR, rms, and rmda.

The study was approved by the Research and Ethics Committee of HUSI and Pontificia Universidad Javeriana (FM-CIEI-0686-21). The Institutional Review Board (IRB) approved each participating institution’s participation in the WHO Global Clinical Platform (11), whose data serve as the basis for this study. Informed consent was waived due to the retrospective design. Data were anonymized, with access restricted to designated investigators.

A total of 7,973 out of 9,050 patients from the Colombian cohort of the WHO Global Clinical Platform for COVID-19 met the inclusion criteria (see Supplementary material F1). In the study population, 58.3% were men, and the median age was 60.7 years (IQR: 49–80). At 30 days, the median age was 59.6 years (IQR: 48.3–80.1) and 70.6 years (IQR: 60.1–81.6) among survivors and non-survivors, respectively.

The most frequent underlying condition was hypertension (29.5%, n = 2,353), followed by obesity (27.6%, n = 2,202) and diabetes mellitus (16.9%, n = 1,346). Among non-survivors, having two or more comorbidities was common (39.3%, n = 354) (see Table 1). Overall, 95.7% (n = 7,627) of patients required hospitalization for more than 24 h, whereas 13.4% (n = 1,072) were admitted to the ICU. The median hospital stay was 6 days (IQR: 3.2–11.4).

At 30 days post-admission to the emergency department, the cumulative incidence of death was 11.3% (n = 901). The Kaplan–Meier survival curves, stratified by NEWS2, CURB-65, qSOFA, and 4C score categories, demonstrate significant differences in survival probability across risk groups (log-rank test, p < 0.0001). For all models, patients in the highest-risk category experience a decline in survival within the first days of follow-up. In contrast, those in the lowest-risk category maintain high survival probabilities over time (see Supplementary material F2).

The discriminative performance of the models was assessed using the AUC, yielding values of 0.69 (95% CI: 0.67–0.70) for CURB-65, 0.67 (95% CI: 0.66–0.69) for CRB-65, 0.63 (95% CI: 0.61–0.65) for NEWS2, 0.62 (95% CI: 0.61–0.64) for qSOFA, and 0.71 (95% CI: 0.70–0.73) for the 4C mortality score. Figure 1 presents the ROC curves for the evaluated models.

In terms of calibration, the GiViTI calibration belt analysis indicated that none of the models achieved adequate calibration (Figure 2). Additionally, the likelihood ratio test (p < 0.05) rejected the null hypothesis of no differences between predicted and observed probabilities for all models.

An observed-to-expected event plot stratified by predefined risk groups was also generated (Figure 2). The results showed that the CURB-65 and CRB-65 models underestimated mortality risk across all predefined risk groups. The NEWS2 model underestimated mortality risk in the low-to-intermediate risk groups. The qSOFA model underestimated the risk in the low-risk group. Similarly, the 4C model underestimated mortality risk in the lower-risk groups and consistently overestimated it in the remaining groups. Detailed information on the comparison of observed and predicted event probabilities for all models is available in Supplementary Table 1R.

When comparing the performance of the 4C and NEWS2 models before and after the introduction of corticosteroid therapy, or before and after the implementation of COVID-19 vaccination in Colombia, we did not observe clinically meaningful differences in overall discriminative ability. For the NEWS2, no significant differences in calibration were found between the periods. However, the 4C model did show better calibration during the period prior to than during the period following the introduction of corticosteroids. Once there was an effective therapy, the model overestimated the risk of death, especially among those at high risk. Detailed performance metrics for each period are presented in Supplementary Figures F3–F6.

The overall performance of the 4C model is presented in Table 2. Given that the 4C model showed the best discrimination among the evaluated models, we recalibrated it and reassessed its performance.

The recalibrated 4C model is of the form: New model = intercept_new (−1.049) + overall slope (0.781)* original 4C model logit. This was used to adjust the coefficients of all variables in the logistic model. The original and recalibrated coefficients, along with the performance of the 4C Mortality Score, are presented in Table 2. An online calculator based on the recalibrated model is available at: https://github.com/Mars1971/Recalibrated-4C-Mortality-Score-for-COVID-19-in-Colombia.

The updated model achieved an observed-to-expected event ratio of 1.00 (95% CI, 0.94–1.07), an intercept of 0 (95% CI, −0.07 to 0.07), and a calibration slope of 1 (95% CI, 0.91–1.09) (Table 2). Calibration was also confirmed via a calibration belt plot, and a non-significant likelihood ratio test (p = 0.644), indicating no meaningful difference between observed and predicted events (Supplementary Figures F7–F9). Discrimination ability slightly improved (DeLong’s test <0.001), as did overall model performance.

The original and recalibrated 4C models demonstrated a net benefit for guiding hospitalization decisions at a mortality risk threshold of approximately 4% and maintained this benefit across the proposed threshold range. At a 4% mortality risk threshold, the cost–benefit ratio was 1:25, indicating that accepting approximately 25 potentially unnecessary hospital admissions may be justified to avoid missing one patient at high risk of death (Figure 3). A detailed table presenting net benefit values across different mortality risk thresholds is provided in Figure 3. The curves with confidence intervals are shown in the Supplementary Figure F10.