We conducted a retrospective analysis of patients' clinical data from diagnostic or therapeutic ERCPs at General Hospital of Ningxia Medical University from May 2022 to June 2023. After applying rigorous inclusion and exclusion criteria, we included a total of 688 cases in the training and validation sets. Eligible patients were those with relevant biliary-pancreatic diseases indicated for ERCP, who had provided written informed consent for the procedure and had granted verbal or written permission for postoperative examinations. Patients were excluded if they presented with acute pancreatitis, had a history of previous ERCP or gastrointestinal reconstructive surgery. Patients that were under 18 years-old (6), had incomplete medical records or surgical videos, did not complete necessary follow-up examinations in a timely manner, did not undergo a full cannulation attempt and abandoned the surgery, or were treated by an endoscopist with less than 50 cannulate procedures (7), were also excluded (Figure 1).

The primary endpoint was established according to current international guidelines and consensus on the incidence of PEP. The criteria for the occurrence of PEP were defined as follows: (1) New onset of abdominal pain was consistent with pancreatitis (acute persistent upper left abdominal pain); (2) Serum amylase levels were three times greater than the upper limit of normal within 24 h after the procedure; (3) Imaging evidence of pancreatitis such as peripancreatic fluid extravasation, pancreatic gland enlargement due to edema, pancreatic duct dilation, pancreatic tissue necrosis, or formation of a pancreatic pseudocyst (two out of these three criteria must be met by the Atlanta Consensus 2012) (7, 8). Failed cannulation was defined as the inability to correctly enter the bile or pancreatic duct despite all techniques and efforts (9). Difficult cannulation was characterized by a cannulation time exceeding 5 min, more than five papillary contacts, or incorrect entry into the pancreatic duct more than once by ESGE Guidelines 2019 (2, 10). We further monitored preoperative serum calcium ion levels, classifying them using our hospital's upper limit of normal value at 2.12 mmol. In this study, the threshold of patient's age was set at <60. The biliary brush cytology sampling methods included cytological brushing or forceps biopsy (11). “Failure to clear bile duct stones” was determined as stones that were not retrieved or not completely retrieved following the full cannulation process. The patient retained a nasobiliary drain at the conclusion of the procedure (12).

To ensure the stability of the results, patients with missing video recording data were excluded (n = 142), which did not affect the results of this study. As the incidence of patients with PEP at our institution was approximately 11%, there was an imbalance in the proportion of patients with and without PEP. To address the imbalance in sample sizes at different levels of the outcome variable, we used the Adaptive Synthetic Sampling (ADASYN) algorithm. This approach was used to survey patients with PEP and normal samples accordingly, achieving a 1:1 match with patients who did not develop PEP postoperatively (13).

Prior to oversampling with ADASYN, we performed data separation by randomly dividing the dataset into two subsets (training and test) with a 7:3 ratio. These subsets were used for modelling and validation, respectively. To minimize the overfitting risk introduced by ADASYN oversampling, we implemented a robust validation strategy. Specifically, we employed 10-fold cross-validation during the model training and tuning phase. Furthermore, the final model was evaluated on a completely independent test set that was not involved in either the oversampling or cross-validation processes. Feature selection within the training set was conducted by a stepwise regression approach grounded in logistic regression with backward elimination (14). Covariance test showed no significant abnormality. This procedure was implemented using the Mass package in R software (version 4.2.2). Variables were selected according to the principle of minimising the Akaike Information Criterion (AIC) and a threshold of p-value < 0.01. This process aimed to identify a subset of features that provided an optimal balance between power and dimensionality (15).

The dataset was subjected to both linear and non-linear ML models, including logistic regression (LR), support vector machines (SVM), gradient boosting trees (GBT) and neural network models. The SVM was implemented using both Linear and Radial Basis Function (RBF) kernels, the GBT was implemented using XGBoost (eXtreme Gradient Boosting) and LightGBM (Light Gradient Boosting Machine), while the neural network model was implemented using a Multi-Layer Perceptron (MLP). The predictive performance of all models was evaluated using five metrics: Area Under the Receiver Operating Characteristic Curve (AUROC), precision, recall, F1 score and accuracy. Finally, ML models with acceptable performance were selected based on the AUC curve.

Categorical variables were assigned numerical codes, and categorical variables were assigned numerical codes, and ization was conducted using EXCEL 2018. Python was used for data processing and model development, relying on the scikit-learn library for the construction of machine learning algorithms. Data statistics, variable selection and construction of conventional LR models were performed using R software (R4.2.2), with a p-value < 0.01 indicating statistical significance. All authors of this article had access to the research data, participated in data collection, reviewed and approved the final manuscript.

A total of 1,748 patients who underwent ERCP procedures met the criteria for this study. Of these, 514 patients with pre-operative acute pancreatitis (hyperamylasemia) were excluded. Further 396 patients with previous ERCP or gastrointestinal reconstruction history were also excluded. A further 150 patients were excluded because the procedure was abandoned or performed by inexperienced endoscopist, the medical records or video data were incomplete. Each exclusion criterion was counted only once. Therefore, 688 patients were included in the final analysis. The study flow chart was shown in Figure 1. Table 1 showed the baseline characteristics of the study population, focusing on the variable PEP. The distribution of key characteristics, including sex, age, presence of diabetes, coronary heart disease, hypertension, history of biliary surgery, cholecystectomy, acute pancreatitis, Ca + and total bilirubin levels, and various clinical procedures and interventions, were reported. We then performed the PCA dimensionality reduction plot for ADASYN and found on difference between PEP (happen) and non-PEP (Not) patients (Figure 1B).

Based on the characteristic sets listed in Table 1, we used a stepwise regression method with logistic regression and backward elimination to select variables for feature filtering. Seven characteristic sets were included in the features with smaller AIC curves in the training set and a p-value < 0.01: Periampullary diverticulum (PAD), pancreatic stent placement (PSP), Pancreatic injection (PI), number of guidewire passages into the pancreatic duct (NGP), dilatation of extrahepatic bile ducts (DEBD), age, and coronary heart disease (CHD). Based on the logistic regression analysis results table (Tables 2–4), the conventional LR model predicts the probability of pancreatitis using the following equation:P=exp(0.154+1.767∗PAD−0.624∗PSP;+0.567∗PI,+1.075∗NGP−0.383∗DEBD−0.639∗CHD−1.176∗Age)/(1+exp(0.154+1.767∗PAD−0.624∗PST+0.567∗PI+1.075∗NGP−0.383∗DEBD−0.639∗CHD−1.176∗Age))

To enhance the interpretability and clinical reliability of our model, we conducted a comprehensive Shapley additive explanations (SHAP) analysis and calibration analysis. For the five models, the SHAP summary figures intuitively showed the contribution of each feature to the model prediction, and highlights the most influential clinical variables (Figure 2). After that, we chose the intersection features selected by LASSO regression and stepwise methods for modeling, and finally removed the pancreatic injection (PI) feature from the equation. The final prediction equation was:P=exp(−0.226+1.739∗PAD−0.617∗PSP+1.235∗NGP+0.43∗DEBD−0.601∗CHD−1.128∗Age)/(1+exp(−0.226+1.739∗PAD−0.617∗PSP+1.235∗NGP+0.43∗DEBD−0.601∗CHD−1.128∗Age))

At the same time, we considered the CHD patients taking aspirin. Among 72 patients diagnosed with CHD, 50 patients (69.4%) regularly took aspirin before ERCP surgery, and only one patient (2.0%) had PEP incidence. For the 22 non-aspirin group, the PEP incidence rate was 18.2% (4/22), showing significant difference compared with aspirin-taken group (p-value = 0.024), indicating that aspirin effect on PEP incidence for CHD patients.

The predictive performance that was measured by the receiver operating characteristic (ROC) model by comparing to the conventional logistic regression (LR) model, was depicted (Figures 3A–F). Models with an area under the ROC curve (AUC) exceeding 0.75 were considered as acceptable. This criterion was met by the following models: LR (Figure 3A) with a test AUC of 0.775 [95% confidence interval (CI) 0.727–0.823]; LR (ML) (Figure 3B) with a test AUC of 0.788 (95% CI 0.7347–0.8408); support vector machine (SVM) (Figure 3C) with a test AUC of 0.812 (95% CI 0.7612–0.8624); XGBoost (Figure 3D) with a test AUC of 0.840 (95% CI 0.7955–0.8851); LightGBM (Figure 3E) with a test AUC of 0.807 (95% CI 0.7575–0.8574); and the multilayer perceptron (MLP) model (Figure 3F) with a test AUC of 0.835 (95% CI 0.7887–0.8822). A detailed performance description for each ML model is supplemented in Table 5.

Then we performed calibration analysis for each model, the calibration curves indicated that Logistic Regression (LR) demonstrated nearly ideal calibration, while LightGBM and Xgboost also showed reasonable calibration performance. In contrast, the calibration curves for SVM and MLP exhibit minor deviations from the ideal line, which remain within acceptable limits and do not substantially affect the clinical interpretability of the predictions (Figure 4A). Finally, we included a decision-curve analysis (DCA) to evaluate clinical benefit at different thresholds. This analysis allows for a direct comparison of the net clinical benefit across a range of threshold probabilities, providing a crucial perspective on the model's value in clinical decision-making beyond traditional discrimination metrics (Figure 4B).

Finally, we developed a simplified, points-based risk scoring system for potential bedside application based on the relative SHAP importance using the LightGBM model, which showed higher AUC score than other models. Using a pre-specified threshold of 3.46 points (derived from the risk distribution in our cohort), patients can be stratified into a Low-Risk group (score ≤3.46) and a High-Risk group (score >3.46) (Figure 5A). This simplified model, despite its ease of use, retained a clinically acceptable discriminative ability with an AUC of 0.75 on our validation set (Figure 5B).

In this study, we developed a predictive model for post-ERCP pancreatitis using ML techniques, employing stepwise regression for feature selection. Based on stringent inclusion and exclusion criteria, we independently identified six factors showing highest correlation with outcomes. We consider periampullary diverticulum, pancreatic duct stent placement, more than one guidewire passage into the pancreatic duct, non-dilated extrahepatic bile duct, age, and coronary heart disease as the most significant clinical variables affecting the risk of PEP. In subsequent ML model development, all ML models outperformed the conventional LR model, with Support Vector Machines (SVM), Gradient Boosting Trees (GBT), and Multi-Layer Perceptron (MLP) models achieving acceptable or superior performance. Figure 6 depicts the overall research process.

Post-ERCP pancreatitis (PEP) represents one of the most frequent and severe complications associated with ERCP (16). This complication can substantially diminish patients' quality of life, augment healthcare expenditures, and in its gravest manifestations, result in patient mortality (17). Although a multitude of predictive models have been developed that demonstrate adept performance, their implementation is often impeded by their complexity, which hinders their integration into routine clinical practice. Furthermore, there may be an insufficient level of recognition among younger endoscopists regarding the potential risk of PEP occurrence. Therefore, it is necessary to develop a prediction model with acceptable accuracy and relatively straightforward predictors for clinical use. In 2021, Koichi Fujita et al. created a practical scoring system for PEP based on a multicenter study in Japan. Considering the weight of seven predictive factors, they developed a model with an acceptable fit and accuracy (AUROC = 0.791) (18). In 2022, Chan Hyuk Park et al. proposed a model based on high-risk factors before and after ERCP but failed to validate the effectiveness of their preoperative risk prediction model (19). Recently, our group used a logistic regression-based backward algorithm to select predictive factors for the establishment of traditional and machine learning models for PEP. Our machine learning model showed better performance and prospects for clinical application than previous models. Traditional multivariate regression is susceptible to small sample bias, particularly in the context of low probability events such as PEP. This approach may also exhibit limited generalization capabilities when addressing complex nonlinear relationships and instances where positive event samples are scarce. To circumvent these constraints, employing distinct ML algorithm-based models using preoperative and postoperative data can potentially enhance predictive performance and surpass the limitations inherent in existing models (20).

ML models have been widely applied in the medical field due to their advantages in handling complex nonlinear relationships and high-dimensional data (21). Substantial literatures have discussed the establishment of ML models related to acute pancreatitis. Our group also paid attention to the work published by Livia Archibugi and colleagues in May 2023, which may be the first discussion on using ML to predict the risk of PEP. They innovatively used the SHapley Additive exPlanations (SHAP) method to open the “black box” of algorithms and study how each feature contributes to the model. Unfortunately, neither traditional nor ML models showed AUROC curve values that could be referenced and applied clinically (Gradient Boosting = 0.67, LR = 0.56). They suggested that this might be related to insufficient model training due to data imbalance caused by the low incidence rate (6%) of PEP (22). After augmenting the training set data using the ADASYN method, all models satisfactorily predicted the occurrence of postoperative pancreatitis in the validation set, with XGBoost showing the best AUC: 0.840 (95% Cl 0.7955–0.8851). Meanwhile, the above results also need to be further validated by statistical comparison such as DeLong test using bigger sample size in future. We recognize that ADASYN may induce the risk of overfitting, leading to forecast deviation. Meanwhile, it can indeed effectively improve the performance of classifiers and neural networks when the dataset is highly imbalanced; therefore, it is feasible to use these data to predict PEP events. That is, endoscopists can achieve dynamic prediction based on these features and decide on perioperative strategies, allowing nursing decisions to be more precise and personalized.

After ERCP, the interpretation and communication of ML models pose a challenge due to the existence of algorithmic “black boxes”. In this investigation, a model-agnostic interpretive approach was employed to discern the underlying clinical elements that may contribute to the onset of post-endoscopic pancreatitis in patients. Using a sequence backward stepwise regression algorithm, we are instrumental in predicting post we identified key clinical variables that are instrumental in predicting PEP. These variables include the presence of periampullary diverticulum, placement of a pancreatic duct stent, the frequency of guidewire cannulation into the pancreatic duct, non-dilated extrahepatic bile ducts, patient age, and the presence of coronary heart disease. Notably, pancreatic duct stent placement, guidewire passages, non-dilated extrahepatic bile ducts, and patient age are all recognized as either risk or protective factors according to the guidelines set forth by the European Society of Gastrointestinal Endoscopy (ESGE) or the American Society for Gastrointestinal Endoscopy (ASGE) (7, 10, 23). NGP has been used as an independent risk factor for PEP in one recent study (24). In another study, the non-dilated extrahepatic bile ducts factor was found to be a significant predictor for PEP using the comprehensive systematic review and meta-analysis method (25). Periampullary diverticulum may be somewhat controversial. From the data collected, most of the patients with periampullary diverticulum (PAD) included in our study had intra-diverticular papilla or a papilla less than 2 cm from the diverticulum. These diverticula could potentially affect the sphincter of Oddi, which normally serves as the main valve for the pancreaticobiliary tract. In patients with PAD, the presence of a diverticulum often causes the papilla to lose its normal morphology partially or completely, leading to variations in the direction of the pancreaticobiliary ducts (26–28). Additionally, the lack of duodenal smooth muscle support at the site of the diverticulum can lead to reduced tension in the distal walls of the pancreaticobiliary ducts, diminished emptying capacity, and post-sphincterotomy tissue edema exacerbates this condition (29). A compelling piece of evidence is that sphincter of Oddi dysfunction (SOD) has been identified as an important and independent risk factor for PEP in most studies (10). However, regarding coronary heart disease (CHD), current guidelines do not indicate it as an independent protective or risk factor for PEP. Patients with CHD are often on long-term medications, including aspirin. Yet, in our data, the group with CHD exhibited a lower incidence of PEP, aligning with the findings of a study published by Harsh K. Patel et al. They compared 1,374,773 ERCP procedures and found a lower incidence of PEP in patients with a history of myocardial infarction (MI) or coronary revascularization surgery (PCI or CABG) (14.1% vs. 15.4%, p < 0.001), though they did not discuss the reasons for this finding in their conclusion (30). In many cases, patients undergoing urgent ERCP for conditions such as common bile duct stones or suppurative cholangitis may not switch to heparin bridging therapy in a timely manner. It is speculated that the lower incidence rate of PEP observed in this patient group could be attributed to the protective effect of nonsteroidal anti-inflammatory drugs (NSAIDs) administered preoperatively to those with CHD against PEP. This reflects the interaction between model variables, and since the Backward selection algorithm is generally more flexible in variable selection, considering more interactions and nonlinear relationships, it is better equipped to capture the complex influences and obtain solid association between CHD and other variables in future (31). It also reminds us that when assessing a patient's PEP risk, it may be necessary to consider the impact of medications routinely taken by the patient on PEP, which could affect the choice of preventive measures and treatment strategies (32).

Amidst the extensive discourse on the application and dosage of NSAIDs and other potential prophylactic medications for PEP, as proposed by various clinical guidelines—including those from ESGE, ASGE, and the Chinese Guideline for ERCP (2018 Edition)—it is imperative to consider the evidence underpinning these recommendations. it is questionable how many patients undergo a comprehensive PEP risk assessment preoperatively (2, 10, 33, 34). Therefore, the development of a streamlined predictive model for this population is worthy considering for endoscopists. Initially, a thorough evaluation based on the patient's physical condition and radiological findings should be conducted by endoscopists preoperatively. Patients with pre-existing CHD often took aspirin, which, although found to reduce the incidence of PEP in our data, increases the risk of bleeding due to the continued use of anticoagulants when the sphincterotomy is performed during the ERCP procedure (3, 19). Furthermore, if a periampullary diverticulum is discovered during the procedure, particularly Li-Tanaka type I or II diverticula that severely affect the papillary morphology (35), the relationship between the opening of the papilla and the axis of the bile duct should be carefully considered to choose an appropriate cannulation method to reduce the incidence of postoperative complications (36). In this study, we have identified some unique factors for early identification of PEP, which can provide guidance for future multicenter clinical trials. In future multicenter experiments, with independent validation and more case data, a scoring system for PEP can be established. Based on the confirmed model, we could develop an online risk assessment tool in the future to estimate the risk of PEP in ERCP patients (37). In summary, if patients present with the high-risk factors mentioned above, clinicians should take corresponding measures to improve patient outcomes and expedite discharge.

This investigation exhibited several notable strengths that warrant emphasis. Firstly, the study implemented a rigorous modeling procedure that encompassed stringent criteria for inclusion and exclusion, exhaustive data processing, meticulous feature selection, model construction, and assessment. This methodological approach yielded a predictive model characterized by streamlined factors yet relatively practical outcomes. Meanwhile, a checklist of reporting guidelines such as transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) should be performed to improve reproducibility (38). Secondly, this study applied ML methods for the first time to validate the filtered model features, offering higher predictive accuracy than traditional linear models even when dealing with complex data patterns and associations. However, there are several limitations to acknowledge. Primarily, this study utilized a retrospective single-center cohort study due to the limitation of time and resource, limiting the generalizability of this model to other regions due to the differences in patient demographics or procedural practices. Despite the strictest inclusion and exclusion criteria, the small sample size necessitates prospective observational studies for more rigorous validation. The smaller sample size also poses the risk of overfitting in ML models. Secondly, due to the low incidence of PEP postoperatively at our institution, there is a significant imbalance between the samples of PEP patients and non-PEP patients. For model evaluation, precision, recall, and other metrics of the predicted model at different thresholds should be reported besides the AUC assessment. This study relies on stepwise logistic regression for feature screening, which may miss prediction patterns that can only be recognized by nonlinear models (such as high-order interaction terms or threshold effects). Although the final XGBoost model can still partially compensate for such omissions through tree structure, future research should explore ML methods based on regularization to better adapt to the needs of complex models. The interpretability tools for ML models such as SHAP or LIME also should be considered to confirm the model's clinical decision-making basis. Meanwhile, the information about drugs usage was not fully considered in this study because of the limitation of clinical information of patients. Although an adaptive synthetic sampling algorithm is used to balance the sample sizes, external validation is still required in future research. In summary, these limitations call for a large-scale and multicenter cohort study in future to validate this model.