In this retrospective cohort study, we included hospitalized patients admitted for the diagnosis of COVID-19 using the Healthcare Cost and Utilization Project (HCUP) State Inpatients Database (SID) of Nevada, Florida, Michigan, Maryland, and Kentucky in 2020. The HCUP SID is a longitudinal database containing the inpatient discharge records of state hospitals. The requirement for informed consent was waived because the HCUP SID was de-identified. All HCUP data users completed an HCUP Data Use Agreement. Model development, validation and reporting followed the guidelines of the Transparent Reporting of a Multivariable Prediction Model for Individual Prediction or Diagnosis (TRIPOD) (12). This study was approved by the Ethics Committee of the Naval Medical University (No. 2021LL024).

Patients hospitalized with an admitting diagnosis of COVID-19 were included in this study. The COVID-19 diagnosis was identified according to the International Classification of Disease 10th revision (ICD-10) code (U071) (13). Patients aged < 18 years or those with a length of stay < 48 h, were excluded. Missing data were observed for three variables: age (2 patients), sex (7 patients), and clinical outcome (109 patients) (Figure 1). These records were excluded because the percentage of missing values was < 0.06% (117/199 056).

The primary outcome was in-hospital death of patients with COVID-19. This outcome was selected because early identification of patients at a high risk of adverse outcomes can support clinical decision-making to decrease COVID-19-related mortality.

Based on a review of the existing literature to identify factors associated with COVID-19 mortality, the following predictors available at hospital admission were selected to predict in-hospital mortality: age, sex, and Elixhauser Comorbidity Index (ECI) (14, 15). The ECI identified a set of preexisting conditions that were unrelated to the principal diagnosis and significantly impacted resource allocation and in-hospital mortality (16). A total of 38 comorbidities were included in the HCUP Elixhauser Comorbidity Software tool (version 2022.1), which was originally developed using the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM). As the primary intention was to develop an easy-to-use prediction model for bedside use, age was converted into four categories (≤ 60, 60–69, 70–79, or ≥ 80 years) in the final model.

Patients hospitalized in Eastern United States (Florida, Michigan, Kentucky, and Maryland) in 2020 were included in the training set. Considering the rule of at least 10 events per candidate predictor parameter (10 EPP) (17), a sample size of 17,954 in-hospital deaths in the training set was sufficient for 40 predictor variables.

Using 10-fold cross validation, we performed a least absolute shrinkage and selection operator (LASSO) logistic regression for predictor selection to minimize overfitting and potential collinearity of variables (18). LASSO logistic regression was conducted to fit the prediction model for all lambda and to use the one standard error (1SE) rule to select the lambda, which can reduce dimensionality and optimize the prediction model. Variables selected by LASSO logistic regression were subsequently assessed using a standard binary logistic regression (enter method). Finally, a prognostic nomogram for predicting the risk of in-hospital death in patients with COVID-19 was constructed based standard binary logistic regression results.

Discrimination of the prediction model was evaluated using the area under the curve (AUC) of the receiver operator characteristic (ROC). The Youden index was used to determine the optimal cutoff point that maximized sensitivity and specificity. Overall goodness of fit was assessed using the Brier score. A calibration curve was used to examine the agreement between the predicted and observed in-hospital deaths. Decision curve analysis (DCA) was performed to validate clinical utility by calculating the net benefits at different threshold probabilities.

To evaluate the discriminatory performance of the prediction model in different settings, we conducted a sensitivity analysis with complete data that included patients aged < 18 years old or hospitalized for < 2 days. Given the reported association between ethnicity and COVID-19-related death, further sensitivity analysis was performed by stratifying the training and validation sets by ethnicity (19).

Patients hospitalized in Western United States (Nevada) in 2020 were included in the validation set. The prediction model's performance was assessed based on discrimination, calibration, and clinical utility. A sensitivity analysis was also performed with the complete data and stratification by ethnicity of the validation set.

Continuous variables were presented as mean (standard deviation, SD) or median (interquartile range, IQR), and categorical variables as frequencies (%). The demographic and clinical characteristics of the patients were analyzed using the Kruskal–Wallis test, Chi-square test, or Fisher's exact test, as appropriate. The standard binary logistic regression results are reported as coefficients and odds ratios (OR) with 95% confidence intervals (CI). All tests were two-sided, and statistical significance was set at p < 0.05. All statistical analyses were performed using R software (version 4.0.3). The packages used in this study included “glmnet”, “rms”, “pROC”, “calibrate”, “dca.R”, and “lrm”.

A total of 199 056 patients with COVID-19 were screened for eligibility, and 180 714 patients were assigned to the training set (n = 168 137) and validation set (n = 12 577) (Figure 1). The median age was 67 years (IQR, 55—78), ranging from 18 to 109 years old. There were 61 795 (34.19%) patients aged < 60 years old, 40 024 (22.15%) patients aged 60—69 years old, 40 260 (22.28%) patients aged 70—79 years old, and 38 635 (21.38%) patients aged ≥ 80 years old. Of these patients, 84 970 (47.02%) were female, 89 052 (49.28%) were of white ethnicity, and 87 235 (48.27%) were of non-white ethnicity. There were 73 076 (40.44%) patients with over three comorbidities and 19 306 (10.68%) deaths during hospitalization. In the training set, the mortality was 11.06% (8683/78 498) in the white ethnic group and 10.17% (8667/85 212) in the non-white ethnic group. Similar results were observed in the validation set, with 11.24% (1186/10 554) mortality in the white ethnic group and 8.21% (166/2023) mortality in the non-white ethnic group.

The most common comorbidity was uncomplicated hypertension (77 656, 42.97%). Other comorbidities such as obesity, complicated hypertension, diabetes with chronic complications, and chronic pulmonary disease, also affected more than 20% patients with COVID-19. Compared to surviving patients, those who died in the hospital were more likely to be male, older adults, and have more comorbidities. The baseline characteristics of the training and validation sets were presented in Table 1.

The prediction model was developed using a training set of 168 137 patients, of which 17 954 in-hospital deaths occurred. In total, 40 candidate variables were screened using LASSO regression, and 15 independent variables were selected for multivariate logistic regression (Figure 2). In-hospital deaths increased with age, male sex, and 13 comorbidities: deficiency anemias, metastatic cancer, congestive heart failure, coagulopathy, diabetes with chronic complications, complicated hypertension, neurological disorders unaffecting movement, obesity, peripheral vascular disease, pulmonary circulation disease, moderate renal failure, severe renal failure, and weight loss. Among these predictors, ORs were highest for those 60—69 years old (OR 2.26, 95% CI: 2.14—2.39), 70—79 years old (OR 3.34, 95% CI: 3.17—3.53), and ≥ 80 years old (OR 4.44, 95% CI: 4.2—4.69) (Table 2). Finally, a nomogram was generated by assigning a weighted score to each selected variable (Figure 3). The calculation of risk score was as follows: risk score = 55 (60–69 years old) + 81 (70—79 years old) + 100 (≥ 80 years old) + 25 (male) + 13 (deficiency anemias) + 36 (metastatic cancer) + 18 (congestive heart failure) + 24 (coagulopathy) + 21 (diabetes with chronic complications) + 8 (hypertension, complicated) + 30 (neurological disorders unaffecting movement) + 16 (obesity) + 15 (peripheral vascular disease) + 14 (pulmonary circulation disease) + 10 (renal failure, moderate) + 23 (renal failure, severe) + 37 (weight loss). The optimal cut-point was 107 (corresponding to a threshold probability 0.102), which divided patients into low-risk group (score ≤ 107) and high-risk group (score > 107).

The prediction model showed moderate discrimination for in-hospital deaths with an AUC of 0.726 (95% CI: 0.722—0.729) in the training set. Maximal discrimination was achieved at a cutoff value of 0.102 which provided a sensitivity of 0.642 and a specificity of 0.697 (Figure 4A). The calibration curve showed a slightly overestimated risk of in-hospital death in the training set (Brier score = 0.090, slope = 1, intercept = 0) (Figure 5A). DCA was conducted to assess the clinical utility of the present model (Figure 6A). When the threshold probability ranged between 3% and 29%, applying the nomogram to determine whether to add an intervention obtained a greater net benefit than treating no patients or all patients.

The validation set consisted of 12 577 patients with 1352 in-hospital deaths. The AUC of the prediction model in the validation set was 0.708 (95% CI: 0.694—0.721), indicating moderate discrimination (Figure 4B). A cutoff value of 0.107 (sensitivity = 0.614, specificity = 0.703) was found to be the optimal level for discriminating in-hospital deaths in the validation set. The calibration curve also demonstrated a slightly overestimated risk in the validation set (Brier score = 0.091, slope = 0.98, intercept = −0.078) (Figure 5B). The DCA showed that when the net benefit of the nomogram was higher than the strategy of “treat all” and “treat none”, the threshold probability ranged between 3% and 26% (Figure 6B).

Sensitivity analysis was conducted using complete case data, which included patients aged < 18 years old or hospitalized for < 2 days. The prediction model showed similar discrimination with an AUC of 0.733 (95% CI: 0.729–0.736) in the training set (n = 185 325) and an AUC of 0.711 (95% CI: 0.699–0.724) in the validation set (n = 13 614). Additionally, the prediction model showed better discrimination in the non-white ethnic group (AUC 0.774, 95% CI: 0.743–0.805) than the white ethnic group (AUC 0.697, 95% CI: 0.683–0.711) in the validation set. Similar results were observed in the training set: an AUC of 0.746 (95% CI: 0.740–0.751) in the non-white ethnic group and 0.708 (95% CI: 0.702–0.713) in the white ethnic group.