In the Surveillance, Epidemiology and End Results (SEER) database, 61036 individuals with bone metastases between 2010 and 2018 were examined for this investigation. The SEER database is a project of the National Cancer Institute and, in an effort to lessen the burden of cancer, it provides an authoritative source of information on cancer incidence and survival rate of 28.0% of the population in the United States. The SEER program is supported by the Surveillance Research Program in NCI’s Division of Cancer Control and Population Sciences (http://seer.cancer.gov). SEER gathers data on cancer cases from a variety of locations and sources in the whole United States. The SEER*Stat Version 8.4.0.1 program was used in this investigation to retrieve information about individuals with bone metastases from the SEER database (2010–2018). CUP Patients were included in analysis. Patients were disqualified if they met the following criteria (1): 18 years old or less (2), death due to missing or unknown cause (3), having missing values in needed variable, and (4) having a follow-up of 3 months or less ( Figure 1 ). Based on the above inclusive and exclusive criteria, 1010 patients were enrolled, and a 6:4 split of the entire patient population was randomly assigned to a training group (n=600) and a validation cohort (n=410). Patients from the validation cohort were used to validate the model, which had been developed with patients from the training cohort. A series of 106 CUP patients with bone metastases underwent external validation. Between December 2013 and June 2022, these patients were gathered from the Peking University First Hospital and the Hainan hospital of Chinese PLA General Hospital. The study protocol was approved by the Ethics Committee of the Peking University First Hospital and the Hainan hospital of Chinese PLA General Hospital, and written informed consent was waived due to retrospective data in nature.

The study extracted the following variables: age (years), race (black or other or unknown or white), sex (female or male), brain metastasis (no or unknown or yes), liver metastasis (no or unknown or yes), lung metastasis (no or unknown or yes), radiation (no/unknown or yes), and chemotherapy (no/unknown or yes). Based on the Extent of Disease classification and the American Joint Committee on Cancer, the tumor stage and node stage were recoded. Three-month mortality was defined as patients had a survival outcome of three or less months according to the data in the SEER database.

This study used four machine learning approaches including random forest (14), gradient boosting machine (15), decision tree (16), and eXGBoosting machine (17) to establish models in the training cohort. Leo Breiman in 2001 introduced random forest, and, using the bagging method (Bootstrap AGGregatING), random forest takes use of a series of decision trees with low reciprocal correlation and randomly selected attributes (14). Gradient boosting machine, introduced by Microsoft, is regarded as an ensemble algorithm, and it is able to provide an efficient use of the gradient boosting algorithm with the primary benefit of dramatic acceleration of the training algorithms (15). Decision tree is a data mining method for classification data, and it is presented as a tree-like structure, and in this structure each internal node indicates a test of a feature and each leaf node indicates a classification (16). eXGBoosting machine is a supervised machine learning model, and it utilizes an improved generalized gradient boosting technique to quickly determine the value of all input features (17).

Prior to modelling, significant variables should be identified by multivariate analysis in the training cohort. In the validation cohort, the nine prediction measures, including area under the receiver operating characteristic (AUROC) curves, discrimination slope, calibration slope, intercept-in-large, sensitivity, specificity, precision, Youden index, and accuracy, were used to evaluate predictive performance of the models. Among the nine metrics, AUROC, calibration, sensitivity, specificity, and accuracy are commonly used to evaluate model’s effectiveness. The optimal model is the one with the most favorable discrimination and calibration.

AUROC was plotted using “pROC” package. Accuracy is better when the area under the curve (AUC) is higher. In details, an AUC value of 0.5 denotes chance, whereas an AUC value of 1.0 denotes complete compliance. Calibration curve was plotted using the “val.prob.ci” function, and the curve was plotted with the predicted probability against actual probability. Decision curve was plotted using the “ggDCA” package, and the curve was used to evaluate the clinical benefit of the model by quantifying the net benefit under different threshold probabilities. In this curve, two reference lines were placed to show the highest clinical cost (treat-for-all plan) and no clinical benefit (treat-for-none plan).

The optimal machine-learning model was used to construct the web-based calculator. Firstly, the optimal model was saved as the format of PKL document via Python software (version 3.9.7). Secondly, the optimal model and corresponding Python code were both uploaded to the GitHub (https://github.com/) website. Lastly, a web calculator can be subsequently deployed after interlinking the Streamlit app (https://share.streamlit.io/) into the GitHub. In the calculator, this study designed a panel of feature selection, an introduction of the web calculator, a submit bottom, and results showing probability of three-month mortality and risk group classification.

All continuous data from the entire study were shown as mean and standard deviation (SD), while all categorical data were shown as proportions. To check for comparability, data from patients in the training and validation cohorts were compared. In the training cohort, a subgroup analysis was done between individuals who experienced three-month mortality and those who did not. Additionally, risk stratification was carried out in the study based on the ideal cut-off value (threshold), and a comparison between low-risk and high-risk groups using Chi-square test. R programming language (version 4.1.2) was used for data visualization and statistical analysis, while Python (version 3.9.7) was used for machine learning operations. A P value of less than 0.05 was regarded as statistically significant, and all P values were two-tailed.

A total of 1010 bone metastasis patients were enlisted for analysis based on the inclusive and exclusive criteria. Age was 71.41 years (SD: 13.43 years) on average. The majority patients were white (79.5%) and male (54.3%) ( Table 1 ). Only a small fraction of patients suffered from brain metastasis (9.7%), but up to 46.4% of patients had liver metastasis, and 35.0% of patients diagnosed with lung metastasis, indicating that the burden due to metastatic diseases was relatively high. The therapeutic interventions were not so commonly performed since only 26.8% of patients underwent radiation and 20.7% treated with chemotherapy possibly. This could be because the primary origin of cancers was not known, making it difficult to undertake the proper interventions. The median survival time was 1.0 months (95% CI: 0.83-1.17 months) in the entire cohort. In addition, Table 1 also demonstrated that the baseline characteristics were comparable between the training and validation cohorts (All P>0.05).

Of all enrolled patients, up to 72.38% patients passed away at or within three months. Patients with bone metastases from CUP saw relatively steady three-month mortality from 2010 to 2018 ( Figure 2A ). The three-month mortality increased significantly with age ( Figure 2B ). When compared to patients without three-month mortality, patients with three-month mortality had significantly a higher age (73.21 years vs. 66.12 years, P<0.001, Table 2 ) and a higher proportion of white people (82.0% vs. 76.3%, P=0.029). Besides, three-month mortality group had a significantly greater rate of liver metastasis (47.8% vs. 40.5%, P=0.031) and lung metastasis (36.3% vs. 26.6%, P=0.035) and a significant lower rate of receiving radiation (21.1% vs. 45.7%, P<0.001) and chemotherapy (9.4% vs. 49.7%, P<0.001) versus no three-month mortality group.

Univariate analysis revealed that age (P<0.001), other race (P=0.044), liver metastasis (P=0.027), lung metastasis (P=0.011), radiation (P<0.001), and chemotherapy (P<0.001) were significantly associated with three-month mortality ( Table 3 ). Based on the multivariate analysis, age (P=0.031), liver metastasis (P=0.008), lung metastasis (P=0.012), radiation (P=0.002), and chemotherapy (P<0.001) were significantly relevant to three-month mortality. To be more specific, older age, lung metastasis, and liver metastasis were risk contributors, while radiation and chemotherapy both were protective features. Depending on the multivariate analysis, features for the model was determined. Thus, the four machine learning approaches were used to train and optimize models using the aforementioned five factors.

A web calculator was constructed according the optimal model (the random forest model) in the study. Visiting https://starxueshu-codeok-main-8jv2ws.streamlitapp.com/, users is able to access to the online calculator. If the online calculator has gone to sleep (shut down), users are able to access to it via clicking “Yes, get this app back up!”. After about 30 seconds, the web-based calculator would be accessible. Users can choose features according to their conditions in the panel of selecting parameters, and then probability of three-month mortality could be obtained by submitting all parameters. In addition, related therapy recommendation was displayed in accordance with the risk stratification. The risk stratification was achieved based on the best cut-off value (71.10%) in the random forest model: Patients in the high-risk group were 1.99 times more likely to suffer from three-month mortality than patients in the low-risk group (P<0.001, Table 5 ) in the internal validation cohort. External validation showed the similar trend ( Table 6 ): patients in the high-risk group had 2.37-time odds of three-month mortality than patient in the low-risk group (P<0.001). Additionally, introduction of the model was shown at the end of the interface. Supplementary Figure 9 shows a screenshot of the web calculator. In the screenshot, a specific example was presented: a 57-year-old patient without liver and lung metastasis did not receive radiation and chemotherapy, and the three-month mortality was up to 72.30%.

This study found that older age, lung metastasis, and liver metastasis were significant contributors for three-month mortality, with radiation and chemotherapy being protective factors for survival. Furthermore, using machine learning, the study developed an accurate model that was able to predict survival among bone metastases patients from CUP, and its predictive performance showed good discriminative and calibrating ability. The model was incorporated into a web-based calculator to encourage clinical reference and research use. This model might be a useful tool to facilitate personalized survival estimation and do some help to guide clinical decision-making.

In the entire cohort of patients, up to 72.38% patients passed away at or within three months, and this incidence was significantly higher as compared to that among patients with other cancers. It was reported that 33.7% of bone metastasis patients with common cancers suffered from three-month mortality (18) and 44.4% of lung cancer patients with synchronous brain metastasis had early death (a survival outcome of three or less months) (19). Raghav et al. (20) showed that the median survival time of CUP patients was 14.7 months in a retrospective study. Jin et al. (21) found that the median overall survival time of CUP patients was 6.0 months among patients from the SEER database. Chambard et al. (22) reported that bone metastasis patients with lung cancer had a median survival of 7.0 months. Another large retrospective study found that the median survival time of bone metastasis patients with common cancers was 6.0 months (18). As for the cohort of patients in the present study, the median survival time was only 1.0 months. This short survival time could be explained by the evidence that bone metastasis and CUP were all contributors to negatively affecting survival outcome. What’s more, the majority of CUP (80%) had unfavorable prognosis (2). Additionally, this study found that the three-month mortality increased significantly with age, and Shen et al. (19) found the similar trend among lung cancer patients with synchronous brain metastasis.

The model features in the study contained five variables: age, lung metastasis, liver metastasis, radiation, and chemotherapy. These variables were also proved to be associated with survival outcome among CUP patients in other studies (20, 23). Furthermore, a recent study showed that gender, Eastern Cooperative Oncology Group performance status, histology, number of metastatic sites, and neutrophil-lymphocyte ratio were independent prognostic factors for survival among CUP patients (20). Huey et al. (24) demonstrated that high neutrophil-to-lymphocyte ratio were associated with worse overall survival among CUP with bone-predominant or lymph node-only disease. Consequently, some measures to improve empirical chemotherapy or radiation, performance status, and neutrophil-lymphocyte ratio would be beneficial to survival prognosis among CUP patients.

A number of survival scoring systems had already proposed to predict survival outcome among bone metastasis patients (25), spine metastasis patients (9), and various cancer patients (26–28). As for CUP, in the year of 2021, Raghav et al. (20) developed a model to predict survival prognosis among CUP in a series of 521 patients and validated the model in two cohorts (n=103 and 302). Five independent prognostic factors were included in the model and the model had a C-index of 0.71. In the same year, Jin et al. (21) proposed a model for predicting survival among CUP. A total of 3347 patients were divided into a training cohort and a validation cohort. The C-index of the model was 0.705 in the training cohort and 0.727 in the validation cohort. More recently, in 2022, Yang et al. (29) created a model including six independent prognostic factors (pathology, visceral metastases, Frankel score, weight loss, hemoglobin, and serum tumor markers) to predict survival outcome among spinal metastasis from CUP in a retrospective derivation cohort of 268 patients and this model was validated in a prospective validation cohort of 105 patients. The C-index was 0.775 in the derivation group and 0.771 in the validation cohort. The above prediction models were designed for CUP and might not be applicable in particular bone metastasis patients with CUP. In the present study, the C-index was up to 0.796 based on the random forest model, and the number was the highest as compared to the above studies. Internal and external validation both confirmed that the model had favorable discriminative and calibrating ability.

Risk classification of patients was accomplished in the study, and patients could be split into two risk categories based on the ideal threshold, allowing for the personalized execution of therapeutic strategies. Patients in the high-risk categories had a roughly two-fold greater chance than those in the low-risk groups of dying within three months.

For palliative pain relief, patients in the high-risk group may benefit from the best supportive care, short-term radiation, or even minimally invasive procedures like cementoplasty (7). The study’s proposed model, which dose not ask for any additional staff training, can be used clinically to forecast the survival benefit of patients with bone metastases and raise the performance of oncologists and radiologists who aren’t professionals to that of experts.

Although this study was well designed, there were still several drawbacks. To begin with, some variables, such as comorbidity and laboratory data, were not available due to the limitation of the SEER database, and clinically the detailed information on cancer were also unavailable because of the unclear primary cancer site. Incorporating those variables might further improve prediction performance of the model, but the AUC values demonstrated that the model was useful enough to predict three-month mortality. Furthermore, our study aimed at proposing a model with routine clinical data that were widely available and easily accessible. Under such circumstances, it would be more practical for oncologists to apply the model in these situations. In addition, external validation was only performed in a small study, thus the generalization of the model needs further validation. Therefore, although the model was validated and embraced favorable prediction performance, it warrants further extensive revision and validation.

The random forest model has promising performance with favorable discrimination and calibration. This study suggests a web-based calculator based on the random forest model to estimate the three-month mortality among bone metastases from CUP, and it may be a helpful tool to direct clinical decision-making, inform patients about their prognosis, and facilitate therapeutic communication between patients and physicians. Patients in the high-risk group may better be treated with best supportive care due to very limited survival expectancy.