This study utilized data from the ED electronic medical records of Shanghai East Hospital affiliated with Tongji University, a large tertiary teaching hospital with an advanced information management system. The dataset included all ED records from 1 January to 31 December 2023. Inclusion criteria comprised adult patients with medical ED visits. Exclusion criteria included pediatric cases, surgical cases, and index ED visits resulting in hospital admission, death, transfer to other facilities, or departure against medical advice. To minimize potential confounding from planned short-term revisits, patients with revisit intervals of 3 ≤ 𝑡_subsequent − 𝑡_initial ≤ 7 days were excluded. The remaining patients were categorized into two groups based on their revisit behavior following ED discharge. Those who did not return within 7 days were classified as the control group, while those who revisited the ED within 72 h of discharge (𝑡_subsequent − 𝑡_initial < 3 days) were classified as the URV group. For patients with multiple qualifying URV episodes during the study period, only the first occurrence was included. Variables extracted from the initial ED visit were grouped into four categories: demographic information, clinical characteristics, behavioral patterns, and hospital-specific details. All predictor variables were documented during the initial ED evaluation and were available before the discharge disposition decision. A comprehensive list of these variables, their definitions, and timing of availability is provided in Supplementary Table S1.

The dataset was partitioned using stratified random sampling based on URV status to ensure proportional representation in both training (70%) and testing (30%) cohorts. This stratification procedure was implemented to maintain consistent URV distribution across the partitioned datasets. Standardized mean differences were calculated to assess the balance of baseline characteristics between cohorts, with values below 0.1 considered indicative of adequate distribution equilibrium.

Data preprocessing followed a standardized four-step pipeline. First, missing data were handled using complete case analysis, excluding patients with missing values for any predictor variables. Second, variable encoding was performed: binary variables were processed through label encoding (0/1), ordinal variables (triage levels) were encoded using ordinal encoding to retain hierarchical structure, and the binary outcome was defined as 1 for URV group and 0 for control group. Third, continuous variables were standardized using z-score transformation. Fourth, class imbalance in the training dataset was addressed through a hybrid sampling approach combining ADASYN oversampling with TomekLinks undersampling to achieve balanced class distributions. All preprocessing steps were implemented using Python 3.11.4 with random seed 42 for reproducibility.

To develop predictive models for URVs, four established supervised ML algorithms including LR, RF, SVM, and XGBoost, along with the DL model TabNet, were utilized. Model selection was guided by suitability for tabular clinical data and interpretability requirements for ED deployment. Traditional ML algorithms provide transparent decision pathways essential for clinical acceptance, while TabNet was selected as the DL representative due to its specific architecture for structured data processing through sequential attention mechanisms. The optimal hyperparameter settings were determined through Bayesian optimization coupled with 5-fold cross-validation, with 50 optimization trials conducted per model. During the optimization process, the hyperparameter configuration that achieved the minimum validation loss was selected as the optimal setting, as this criterion indicates the best model generalization capability and minimizes potential overfitting. In the TabNet implementation, training was terminated when no validation performance improvement could be detected across 10 consecutive epochs. After determining the best-performing hyperparameter configurations, the final models were trained on the complete training dataset using the optimized parameters.

Using the independent testing dataset, all models were evaluated through a comparative analysis to identify the one that demonstrated the most reliable performance in predicting URVs. Model performance was assessed based on discrimination, calibration, and clinical relevance. Standardized evaluation metrics were calculated based on confusion matrices. The AI model demonstrating superior overall performance across these multiple evaluation dimensions was selected as the optimal predictor and subjected to additional robustness testing. Four sensitivity analyses were conducted on the optimal model to evaluate performance stability under varied analytical conditions including alternative missing data handling through multiple imputation with chained equations, no-resampling training, temporal holdout validation using final quarter data, and analysis with imaging variables removed.

The optimal performing model underwent systematic interpretability analysis through multiple validation approaches. Feature importance was quantified using model-specific attention mechanisms, with additional model-agnostic validation through permutation importance analysis across 1,000 bootstrap iterations to establish statistical robustness. Multicollinearity assessment among top predictors was performed using correlation analysis. Systematic ablation studies evaluated clinical significance by sequentially removing key predictive features and measuring performance degradation. Subgroup fairness assessment evaluated algorithmic equity across clinically relevant patient subgroups using coefficient of variation analysis to quantify performance consistency.

Comparisons of characteristics between URV and control groups were performed using Mann–Whitney U test for continuous variables and chi-square test for categorical variables. For model evaluation, discrimination was assessed through ROC curve analysis, with the AUC used as the primary metric. Statistical comparisons between model AUCs were performed using DeLong’s test to assess significance of performance differences. Optimal decision thresholds were determined using Youden’s index to maximize overall classification performance, and confusion matrices were constructed at these thresholds to derive performance metrics including accuracy, precision, recall, F1-score and Kappa. Ninety-five percent confidence intervals (95% CIs) were calculated for all key performance metrics to quantify uncertainty. Calibration performance was assessed through calibration curves and quantitative metrics including Brier scores, and calibration intercept, and slope derived from logistic regression of observed outcomes against predicted probabilities. Expected Calibration Error (ECE) was computed as the weighted average of absolute calibration errors across probability bins to provide comprehensive calibration assessment. To determine the clinical relevance of the models, decision curve analysis (DCA) was performed to calculate net benefit across various threshold probabilities.

Figure 1 presents a comprehensive flowchart depicting the data selection process and analytical framework of our investigation. After data cleaning and preprocessing, the final dataset consisted of 143,192 patients, with 24,117 (16.8%) allocated to the URV group and 119,075 (83.2%) to the control group. The initial dataset contained 186,395 ED records. After applying inclusion and exclusion criteria and removing cases with missing data (overall missing rate 6.8%), the final dataset consisted of 143,192 patients, with 24,117 (16.8%) allocated to the URV group and 119,075 (83.2%) to the control group. Variable-specific missing rates were below 1.5% for most variables, with missing data primarily concentrated in blood pressure measurements (8.7%) and initial diagnosis categories (4.3%). Table 1 presents the characteristics of the URV and control groups at the initial ED visit. Due to the large sample sizes in both groups, all examined characteristics exhibited statistically significant differences between the URV and control groups (all p < 0.05). The dataset was further divided into a training set of 100,235 patients and a testing set of 42,957 patients. The proportion of URV cases remained consistent between the training (16.8%, 16,882 patients) and testing sets (16.8%, 7,235 patients), ensuring balanced distribution for model development and evaluation.

To address the class imbalance inherent in the training dataset, we implemented a hybrid sampling approach combining ADASYN oversampling with TomekLinks undersampling techniques. This data balancing strategy yielded a final training set of 148,692 cases, with equal distribution between URV and control groups (74,346 cases each), facilitating optimal feature learning from the minority class.

Five models were developed to predict URV risk, including four ML algorithms (LR, SVM, RF, and XGBoost) and one DL architecture (TabNet). The optimal hyperparameter configurations for each model were identified through Bayesian optimization with 50 trials, selecting configurations that achieved minimum validation loss as the optimization objective (Supplementary Figure S1). The detailed hyperparameter settings that achieved optimal performance can be found in Supplementary Table S2. These optimized configurations were then applied to train the final models.

The predictive capabilities of all models were evaluated on an independent testing set to validate their performance in URV identification. Comprehensive model assessment was conducted through discrimination, calibration, and clinical utility metrics visualized in Figure 2. TabNet and RF demonstrated superior discriminative performance, with both achieving AUROC values above 0.85, significantly higher than the other three ML models (all p < 0.05, DeLong’s test). Calibration performance was assessed through calibration curves and quantitative calibration metrics including Brier scores, intercept, and slope parameters. ECE was computed as the weighted average of absolute calibration errors across probability bins to provide comprehensive calibration assessment (Supplementary Table S3). RF demonstrated the most favorable calibration profile with minimal systematic bias and the lowest ECE, while TabNet showed acceptable calibration performance with slightly higher deviation in the mid-probability ranges. Both models outperformed approaches, with SVM, XGBoost, and LR exhibiting greater calibration errors. DCA across clinically relevant threshold probabilities (10–30%) demonstrated varying model performance patterns (Supplementary Table S4). TabNet exhibited superior and consistent clinical utility throughout the evaluated threshold range. While competing models showed performance degradation at higher thresholds, TabNet maintained sustained net benefit across the spectrum of clinically relevant decision points.

Further analysis through confusion matrices (Figure 3) and performance metrics (Table 2) confirmed the superior performance of TabNet and RF algorithms. Notably, TabNet demonstrated enhanced sensitivity in URV detection with a recall rate of 0.809, failing to identify only 1,379 cases compared to 2,069 cases missed by RF. These results establish TabNet as the optimal predictive model for URV risk stratification.

Four sensitivity analyses evaluated TabNet robustness under varied analytical constraints (Table 3). Multiple imputation analysis incorporating the 6.8% cases with missing data yielded performance metrics comparable to the original complete case approach, with overlapping confidence intervals confirming that missing data handling did not substantially affect model conclusions. No-resampling training demonstrated substantial performance degradation with severe recall decline, confirming the necessity of balanced training for minority class detection in imbalanced medical datasets. Temporal holdout validation using the final quarter (October–December 2023) revealed minimal performance decline despite three-month separation, indicating robust discriminative capability across patient population drift and seasonal variations. Removing imaging variables produced minimal impact, demonstrating preserved predictive performance when restricted to early triage features.

Feature importance analysis of TabNet revealed significant predictive patterns for URV risk stratification through multiple validation approaches. Variable contribution assessment identified ten key predictors quantified through attention mechanisms (Figure 4). Model-agnostic validation using permutation importance with bootstrap stability analysis across 1,000 iterations established statistical robustness, showing consistent importance hierarchies with narrow confidence intervals (Figure 5). Digestive and respiratory system diagnoses demonstrated the highest predictive contributions in both attention-based and permutation-based assessments. Correlation analysis among top predictors revealed no significant multicollinearity (Figure 6), with all pairwise correlations remaining below 0.3, effectively ruling out redundant or artificially inflated feature contributions.

Systematic ablation analysis validated the clinical significance of identified predictors through progressive performance deterioration (Table 4). Sequential elimination of digestive system diagnosis caused the most substantial impact, reducing model accuracy from 0.756 to 0.643 and F1 score from 0.528 to 0.391. Subsequent removal of respiratory system diagnosis, age, P3 triage classification, and ED visit frequency produced incrementally smaller performance decrements, primarily attributed to compromised sensitivity as evidenced by declining recall values.

Based on the identified key predictive variables, subgroup analysis evaluated algorithmic fairness across age, triage level, and initial diagnosis categories (Table 5). TabNet maintained consistent recall performance (0.799–0.830) across all subgroups, ensuring equitable high-risk patient identification. Age-stratified analysis revealed accuracy declining from 0.839 in younger adults to 0.664 in older patients, a 17.5 percentage point differential within clinically acceptable limits. Performance across diagnostic categories (accuracy 0.744–0.896) and triage levels demonstrated variations consistent with clinical complexity expectations. Coefficient of variation analysis confirmed good fairness for diagnostic categories (7.8%) and acceptable fairness for age groups and triage levels (10.4 and 11.2% respectively), supporting equitable clinical deployment.

The accurate prediction of URV risk carries significant implications for healthcare delivery optimization and patient safety enhancement. Our observed URV rate of 16.8% substantially exceeds rates reported from comparable tertiary institutions in regions with established tiered healthcare systems, including Singapore (1.9–2.9%) (5, 11), Canada (8.7%) (3), and the United States (3.6–8.8%) (6, 10). This elevated proportion underscores the critical importance of URV prediction in the Chinese healthcare context, as the developing tiered care system and direct patient preference for tertiary hospitals create a substantial and persistent burden that cannot be rapidly restructured through policy interventions alone.

The 72-h temporal threshold represents the established international standard for URV monitoring, providing optimal balance between capturing clinically significant returns while maintaining administrative feasibility across healthcare systems (21). URVs represent a significant patient safety concern, as studies demonstrate that patients returning within 72 h face substantially elevated risks compared to initial visits, with high-risk URVs requiring hospital admission showing mortality rates of 0.7% and intensive care unit admission rates of 11.1% (22). Among URV patients subsequently requiring ICU care, diagnostic error rates from the index visit may exceed 40%, highlighting the clinical significance of accurate initial assessment and disposition decisions (23). Each URV essentially doubles ED resource utilization (24), with the average visit requiring approximately 4 h of clinical care according to Di Somma et al. (25).

Our finding that P3 patients demonstrated the highest URV rates aligns with international observations that intermediate-acuity patients occupy a clinical vulnerability zone where symptom complexity may be systematically underestimated during initial evaluation, despite subsequently manifesting serious conditions requiring intensive intervention upon return (26). This contrasts with high-acuity patients who receive comprehensive monitoring and low-acuity cases requiring minimal intervention. If these patients could be identified and receive enhanced care planning, including comprehensive discharge instructions, medication counseling, and appropriate follow-up arrangements, many unnecessary returns could be prevented. From a patient-centric perspective, accurate URV prediction enables proactive care planning, potentially reducing adverse outcomes associated with delayed treatment (27, 28). Enhanced discharge planning for high-risk patients may improve treatment adherence, reduce anxiety about symptom recurrence, and strengthen patient trust in healthcare providers through improved discharge safety protocols (29, 30).

Five distinct AI models were evaluated in this study, encompassing both potential linear and nonlinear relationships in URV prediction. Comparative analysis revealed marked variations in model performance. XGBoost exhibited the lowest discriminative ability with an AUROC of 0.621 and poor precision (0.294), indicating considerable false-positive predictions despite high sensitivity. This suboptimal performance may be attributed to XGBoost’s sequential tree-building process, which can be sensitive to noise in clinical data and prone to overfitting when handling complex temporal patterns characteristic of ED visits (31). LR and SVM demonstrated similar discriminative capabilities (AUROC 0.838 and 0.835, respectively), but LR was characterized by poor calibration. This limitation was attributed to the inherent linear relationship assumptions in LR, which were inadequate for capturing intricate patterns in URV risk factors (13). RF and TabNet were identified as superior predictive frameworks, both achieving AUROC values above 0.85. The robust performance was attributed to their architectural advantages in processing tabular clinical data. RF demonstrated effectiveness through ensemble decision trees that capture complex variable interactions, while TabNet utilized self-attention mechanisms for adaptive feature processing (32, 33). Although both models demonstrated strong performance, TabNet exhibited higher recall (0.809 versus 0.714) compared to RF, indicating enhanced capability in identifying potential URV cases. This improved sensitivity was particularly valuable for URV prediction, where identifying high-risk patients for preventive intervention remains a primary clinical objective.

These findings establish TabNet as the best-performing predictive model for URV risk stratification in our comparative analysis. TabNet represents a DL architecture specifically designed for tabular data that employs sequential attention mechanisms and learnable feature selection masks to automatically identify the most relevant predictive variables at each decision step (33). This architectural specialization enables superior performance in processing structured clinical data including demographics, vital signs, and diagnostic codes that are natively stored in electronic medical record systems. Unlike traditional neural networks that process all features simultaneously, TabNet’s multi-step attention process provides adaptive feature importance weighting while preserving interpretable attention weights, making it particularly well-suited for clinical applications where both predictive accuracy and model transparency remain essential for ED implementation. In contrast, LLMs are primarily optimized for natural language processing tasks and excel in scenarios requiring semantic understanding of unstructured text (16). Additionally, the institutional isolation of medical data due to privacy regulations and data governance policies constrains the availability of large-scale multi-institutional datasets that language model training typically requires, presenting practical barriers to LLM deployment in URV prediction contexts where structured tabular data predominates.

Recent investigations of ML-based URV prediction have predominantly utilized traditional algorithms with moderate performance outcomes. A comprehensive 2024 scoping review by Lee et al. (14) analyzed 14 recent implementations and confirmed that most studies remained constrained to conventional ML approaches using structured clinical variables from single institutions. The majority of these investigations focused on 72-h prediction windows using LR, XGBoost, and ensemble voting classifiers, achieving median AUROCs ranging from 0.66 to 0.74 (6, 14, 20, 34). These studies consistently demonstrated a fundamental challenge in URV prediction, specifically achieving optimal balance between sensitivity and specificity. Representative of this limitation, Hsu et al. (20) employed ensemble voting classifiers specifically for abdominal pain patients, achieving AUROC 0.74 with remarkably high specificity (0.89) but severely limited sensitivity (0.39). This pattern of sensitivity-specificity trade-offs appears across URV prediction studies, where traditional ML models either miss substantial proportions of true URV cases or generate excessive false positives that compromise clinical utility (6, 12, 20). To address these methodological limitations, our TabNet implementation utilizing a large-scale dataset of 143,192 patients achieved AUROC 0.867 with sensitivity of 80.9% and specificity of 74.5%, representing a substantial improvement over the reported literature median. This performance enhancement can be attributed to TabNet’s sequential attention mechanisms that enable dynamic feature importance weighting, contrasting with the fixed architectural frameworks that constrain traditional ML models. Notably, this investigation presents the first systematic application of DL architecture specifically designed for tabular data in URV prediction, addressing a critical methodological gap identified in the current literature.

Feature importance analysis of the optimal model revealed several critical predictors for URV risk. Advanced age was identified as a predominant factor, which aligns with multiple previous studies that have documented increased URV risk in older patients, attributed to complex comorbidities, atypical disease presentations, and communication challenges (35–37). Initial diagnoses of respiratory and digestive system diseases were identified as significant risk factors, primarily due to non-specific presenting symptoms and condition complexity. Respiratory conditions often present with symptoms such as cough, dyspnea, or chest discomfort that lack diagnostic specificity and may represent diverse pathologies ranging from benign viral infections to severe pneumonia or pulmonary embolism. Similarly, digestive system diseases frequently manifest through non-specific complaints particularly abdominal pain, which encompasses a broad spectrum of etiologies from self-limited gastroenteritis to acute surgical emergencies, with symptom severity and clinical findings often insufficient to definitively establish diagnosis during initial ED evaluation. These conditions present diagnostic challenges through symptom overlap, multifaceted pathologies, and communication barriers between patients and healthcare providers (38).

The identification of P3 triage classification as a significant predictor differed from previous reports and warrants particular attention (39). This observation indicates that while P1 and P2 patients receive comprehensive evaluation due to the critical nature of their conditions, P3 patients, who constitute a substantial proportion of ED visits, may receive less intensive monitoring despite their complex medical presentations. In the high-volume ED environment, these intermediate-acuity patients might not receive the same level of detailed symptom assessment and continuous observation as higher-acuity cases, potentially leading to missed clinical deterioration or incomplete diagnosis. This finding emphasizes the importance of developing targeted assessment protocols for P3 patients during triage to better identify those at elevated URV risk, particularly given the diverse pathology and symptom complexity often present in this triage category. If high-risk P3 patients identified through predictive models could receive enhanced discharge planning including comprehensive instructions, medication counseling, and appropriate follow-up arrangements, many unnecessary returns could potentially be prevented.

Finally, ED visit frequency in the previous month was identified as a crucial predictor, indicating patient-specific factors including health anxiety, healthcare system navigation difficulties, and treatment plan adherence issues (40). These behavioral patterns highlight the necessity for targeted patient education and support systems to enhance health literacy and care plan adherence.

This study presents notable methodological advances in URV prediction through comprehensive AI model comparison and the novel application of TabNet architecture in emergency medicine. Nevertheless, several limitations should be addressed in future investigations.

First, the single-center retrospective design with an observed URV rate of 16.8% exceeding international benchmarks from tertiary institutions limits generalizability to settings with different patient care-seeking behaviors and healthcare system structures. Additional methodological constraints include potential URV cases at other facilities, URV identification through administrative records rather than direct patient inquiry about the unscheduled nature of their return visits, inability to capture critical psychological factors and physician-patient communication quality that influence return decisions, absence of free-text clinical narratives such as patient chief complaints and physician documentation that might provide additional predictive information, absence of standardized discharge disposition documentation including follow-up recommendations in our electronic medical record system, and restriction to adult internal medicine cases which limits applicability to pediatric and surgical populations with distinct clinical characteristics (41). While the 72-h temporal criterion represents the established international standard, the gold standard for URV identification requires prospective validation through structured patient interviews at the time of return visits to confirm the unscheduled nature of their presentations.

Second, focusing on only the first URV episode per patient may not capture recurrent patterns that could enhance predictive modeling accuracy. The hybrid ADASYN-TomekLinks resampling strategy, while enhancing model sensitivity for minority class detection, generates synthetic samples that may not perfectly reflect true clinical patterns, though our sensitivity analysis without resampling confirmed the necessity of balanced training for effective URV identification. Furthermore, the P1 triage subgroup in our fairness assessment contained only 5 URV cases among 4,453 visits, yielding wide confidence intervals that limit reliable performance estimation for this highest-acuity category.

Third, practical implementation of TabNet in real-world ED settings presents technical and operational challenges that require systematic consideration. The model necessitates adequate computational infrastructure for real-time inference with latency requirements below 5 s to support triage workflow, alongside ongoing maintenance protocols to address potential data drift over time. Integration with existing hospital information systems demands careful attention to workflow compatibility, data security requirements, and compliance with healthcare data protection regulations. Staff training for clinical interpretation of model outputs, change management strategies to facilitate adoption, and technical support infrastructure represent additional barriers to implementation, particularly in resource-constrained healthcare environments. Future prospective deployment studies should systematically evaluate these operational requirements and establish evidence-based implementation frameworks.

Future research directions should focus on three key areas that leverage the clinical advantages identified through our TabNet implementation. First, prospective multicenter studies incorporating real-time patient interviews to validate URV classification accuracy and explore unmeasured behavioral determinants across diverse patient populations are essential. These studies should implement the model within triage protocols to identify high-risk patients based on presenting symptoms, age, triage classification, and recent ED utilization patterns, enabling automated risk assessment that operates alongside standard patient evaluation procedures by processing readily available demographic data, presenting complaints, and vital signs to provide immediate stratification alerts that support discharge planning decisions without disrupting established clinical workflows.

Second, external validation studies across diverse healthcare settings are needed to confirm model robustness and generalizability beyond the current institutional context, with particular attention to how consistent variable coding and standardized data capture across different electronic medical record systems impact predictive performance and clinical utility in real-world ED environments. Such validation efforts would enable health authorities to establish evidence-based URV benchmarks and quality metrics that support systematic ED performance monitoring and resource allocation strategies across healthcare networks. The epidemiological insights derived from consistent URV prediction patterns could inform targeted public health interventions and healthcare system preparedness planning.

Third, integration of this prediction model into ED triage protocols will enable real-time risk assessment to support clinical decision-making and intervention targeting. Future implementations could incorporate free-text clinical data including patient chief complaints and physician narrative documentation to develop multimodal architectures combining TabNet’s structured data processing with LLM analysis of unstructured clinical text, potentially capturing semantic nuances in symptom descriptions and clinical reasoning. Achieving this integration requires future research to prioritize comprehensive data capture systems that preserve both structured variables and free-text narratives, alongside lightweight computational architectures and standardized deployment frameworks that facilitate seamless integration with hospital information systems while maintaining the distinct roles of each AI component within the clinical workflow.